人工性的首次考试：一项以大语言模型为参照的超智能 C 测试

# Artificiality’s First Exam:An LLM-Referenced C-test for Supraintelligence
Source: [https://arxiv.org/html/2610.14005](https://arxiv.org/html/2610.14005)
José Hernández\-Orallo††thanks:Code and data available at[https://github\.com/jorallo/ctest\-one](https://github.com/jorallo/ctest-one)\.Affiliation:Leverhulme Centre for the Future of IntelligenceAffiliation:University of CambridgeEmail:[jh2135@cam\.ac\.uk](mailto:)Affiliation:Valencian Research Institute for Artificial IntelligenceAffiliation:Universitat Politècnica de ValènciaEmail:[jorallo@upv\.es](mailto:)

###### Abstract

Under the accelerated pace of AI development, any benchmark whose difficulty is calibrated with humans or current AI systems becomes circular, or saturates soon\. How can we then evaluate machines that may become more capable than the current machines and the people who write the tests? One answer to the question hinges oninverse problems, inductive inference questions for which difficulty depends on the complexity of the underlying mechanism,giventhe observations and the previous knowledge\. Levin’sK​tKt, a computable variant of Kolmogorov complexity, puts a theoretical upper bound to that difficulty, balancing the simplest representation encoding for the mechanism and the logarithm of the time steps to run that mechanism\. Previous tests derived from Levin’sK​tKt, such as the original C\-test, are based on minimalistic Universal Turing Machines \(UTM\), capable of very limited representations and no previous knowledge\.Reasoninglarge language models \(LLM\) are UTMs that capture a compendium of human culture as knowledge, using prompts as natural language programs\. This paper presents the first test based on Levin’sK​tKtwith LLMs as representational machines\. Fresh test items can be produced at high difficulties, estimated in a ratio scale measured in bits, before any human or frontier model can solve them\. We formalise the measure, devise a lightweight generation and certification pipeline, explore the effect of the power of the reference machine and the gap between examiner and examinee\. We produce a first battery of certified items administered to twelve LLMs\. The resulting agent characteristic curves decrease with difficulty, down to the uncharted high\-difficulty region, thesupra\-intelligence windowof items we can certify, yet beyond generations of frontier intelligence\.

## 1Introduction

We can all conceive and verify questions we cannot solve ourselves\. This possibility is supported by theoretical computer science\([Cook, 1971](https://arxiv.org/html/2610.14005#bib.bib26);[Levin, 1973](https://arxiv.org/html/2610.14005#bib.bib65)\)\. For example, inverse problems\([McCarthy, 2006](https://arxiv.org/html/2610.14005#bib.bib43)\), such as finding the pattern behind a sequence, are hard to solve but easy to verify\. Indeed, verifiability is behind the recent progress of ‘reasoning’ models in areas such as mathematics and software engineering\. But even if we can generate very difficult questions for the next generation of AI systems, it is an open question how to do so systematically, gradating items in terms of an intrinsic measure of difficulty, when no existing human or AI system is still able to solve them\.

As AI progresses, benchmarks devised by human experts or based on current challenges saturate and must be replaced\([Zellers et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib104);[Schlangen, 2021](https://arxiv.org/html/2610.14005#bib.bib105)\); replacements leak into training corpora\([Balloccu et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib81);[Jiang et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib82)\); and progressively ‘harder’ items can no longer be produced and verified\([Center for AI Safety et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib77);[Phan et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib103)\)\. If we want to keep measuring AI as capabilities grow, we need tests that can be*generated on demand*, beyond the reach of the testees, with item difficulty that is*defined by construction*rather than by consensus or norm, and with*certifiable*correctness that does not require the items to be solved directly\.

The goal of this paper is to devise a feasible and systematic procedure to generate problems whose difficulty can be gradated over an intrinsic, non\-moving scale, derived by a weak modelℳ\\mathcal\{M\}to evaluate strong models\. These problems can belong to any broad class of inverse problems, whose difficulty is their conditional Levin’sK​tKtcomplexity\([Levin, 1973](https://arxiv.org/html/2610.14005#bib.bib65);[Levin, 2013](https://arxiv.org/html/2610.14005#bib.bib54)\), which expresses the search effort of Levin’s universal search, an optimal solution \(up to a constant\) to any inverse problem\. The intuitive idea behind the test is simpler: an item is easy if there is a program describing the solutiongiven the observationthat has a low cost and runs fast onℳ\\mathcal\{M\}\.

The scope of this paper is delineated by self\-imposed constraints towards this goal: \(1\) The use of small representation machines, which can fit in small GPUs while still being able to certify items that are unsolvable by frontier models, realising bounded and sparse supra\-intelligence evaluation windows; \(2\) Similarly, the use of a medium\-capability composer, able to govern the certification search procedures; \(3\) An experimental design that is sufficient for the evaluative goals, nothing more nothing less, for sustainability; \(4\) While the new C\-test, ONE, is useful in itself to rank models, it is simply the first of its kind in a series of broader and more sophisticated tests where difficulty can be estimated by a bound to the search cost of the solution given the problem\. Our contributions are:

- •We demonstrate a general method for an examiner that is weaker than the examinee to generate challenging problems that are verifiable \(valid, unquestionable and certified\) and whose difficulty can be anticipated even before any frontier intelligence can solve them, gradating on the upper end of difficulty\.
- •We upend the subjective choice of the universal Turing machine \(UTM\) in the calculation of Levin’sK​tKt, by using LLMs as UTMs, fixing the representations to human culture\.K​tKtbecomes conditional: the logprobs of the promptgiven the observationplus the logarithm of the number of all inference steps\.
- •Models are assigned an ability from zero to infinity, bits of a ratio scale as per measurement theory\. A linear increase in the number ofK​tKtbits corresponds to an exponential growth of search compute, a well\-founded scale to really quantify the pace of AI progress, compared to other populational or contaminated scales\.
- •The item generator is based on the number of retrievals and transformations used in the inversion problem, connecting this generalisation of the original C\-test with the notions of fluid and crystallised intelligence\.K​tKt, however, accounts for simplifications, is very predictive, and is now grounded in human priors\.
- •Verification can be performed with small reasoning LLMs such as Qwen14B or Phi14B, even for very hard items\. The composer is a medium\-sized LLM doing an informed search, avoiding the intractable enumeration dove\-tail approach, previously used for the approximation ofK​tKt\.
- •A 14B\-parameter machine suffices to show strong Spearman correlation betweenK​tKtand item performance of the twelve examinees \(−0\.79\-0\.79\), as well as ability with other psychometric indicators, such as Epoch’s ECI \(\+0\.76\+0\.76\)\. The supraintelligence window is about 190 bits for the strongest examinee, GPT5Sol\.

These are very promising signs for the scalability of supra\-intelligence evaluation\. In the following sections we will explain the properties and results of C\-test ONE, our first exam of an endless series\.

## 2Background

General intelligence evaluation has usually been associated with the identification of patterns that can explain the regularities in data, while at the same time being able to use existing knowledge to solve new problems\. These components roughly correspond to Cattell’s concepts of fluid and crystallised intelligence in humans\([Cattell, 1963](https://arxiv.org/html/2610.14005#bib.bib74)\), but more generally they can be identified as the degree of transformations and retrievals, respectively, in aninverse problem\. Inverse problems are common in education and science\([Groetsch, 1999](https://arxiv.org/html/2610.14005#bib.bib52)\)\. In computer science they are often described as finding the inputxxthat producesyyfor a given functionff, withf⁡\(x\)=yf\(x\)=y\. One general yet very precise class of these problems consist in determining the pattern behind a sequence, common in letter and number series problems in IQ tests\([Thurstone, 1937](https://arxiv.org/html/2610.14005#bib.bib84)\)\. Their most general form consists in inferring the program that generates an observed sequence given a Universal Turing Machine, for which compression is shown to be the best strategy asymptotically\([Solomonoff, 1960](https://arxiv.org/html/2610.14005#bib.bib51);[Solomonoff, 1964a](https://arxiv.org/html/2610.14005#bib.bib49);[Solomonoff, 1964b](https://arxiv.org/html/2610.14005#bib.bib50)\)\. These connections led to the introduction of enhanced tests with compression, such as\([Dowe and Hájek, 1997](https://arxiv.org/html/2610.14005#bib.bib76)\), the C\-test\([Hernández\-Orallo and Minaya\-Collado, 1998](https://arxiv.org/html/2610.14005#bib.bib48);[Hernández\-Orallo, 2000](https://arxiv.org/html/2610.14005#bib.bib56)\)and the ‘universal intelligence test’\([Legg and Hutter, 2007](https://arxiv.org/html/2610.14005#bib.bib71)\)\. For the relation between algorithmic information theory, compression and \(the measurement of\) intelligence more generally, see appendix[S](https://arxiv.org/html/2610.14005#A19)and the extended coverage in[Hernández\-Orallo \(2017\)](https://arxiv.org/html/2610.14005#bib.bib58)\. We just highlight a few key connections next\.

Kolmogorov complexity and Levin search\.For a universal machineUU, the Kolmogorov complexity of a sequencexxis defined asKU​\(x\)≔min⁡\{ℓ⁡\(p\):U⁡\(p\)=x\}K\_\{U\}\(x\)\\coloneqq\\min\\\{\\ell\(p\):U\(p\)=x\\\}withℓ⁡\(p\)\\ell\(p\)being the length of the program \(in bits\)\.KUK\_\{U\}is invariant across universal machines up to an additive constant\([Li and Vitányi, 2019](https://arxiv.org/html/2610.14005#bib.bib66)\), but this constant is as large as coding a machine\-to\-machine interpreter\. Another issue ofKKis that it is incomputable, only being approximated from above\. Levin’sK​tKtcomplexity is defined asK​t​\(x\)≔minp⁡\{ℓ⁡\(p\)\+log⁡τ⁡\(p\)\}Kt\(x\)\\coloneqq\\min\_\{p\}\\\{\\ell\(p\)\+\\log\\tau\(p\)\\\}, penalising the execution timeτ⁡\(p\)\\tau\(p\)of running the program\. The most salient property ofK​tKtis not its computability, but the fact that it quantifies the difficulty of an inversion problem following auniversal search, dovetailing programs by increasing budget of computational steps\. Universal search will solve the problem in a time2ℓ⁡\(p\)⋅τ⁡\(p\)2^\{\\ell\(p\)\}\\cdot\\tau\(p\)at most, which is the best general inversion algorithm despite the huge multiplicative factor\. Taking logarithm of this expression leads toK​tKt\.

C\-test\.The original C\-test\([Hernández\-Orallo and Minaya\-Collado, 1998](https://arxiv.org/html/2610.14005#bib.bib48);[Hernández\-Orallo, 2000](https://arxiv.org/html/2610.14005#bib.bib56)\)generates letter sequences produced by programs in a minimalistic Turing\-complete language \(a RISC computer\), with item difficulty given by the shortest program found by enumeration\. The C\-test introduces the notion of*unquestionability*, i\.e\., no alternative continuation is producible by a program of comparable length\. For instance, the letter series*a, b, c, a, b, c, a, …*is admitted at low difficulty: every short program consistent with the prefix continues with*b*\. The letter series*a, b, b, c, c*is*rejected*: “nth letter n times” predicts*c*, but “oneaa, then each letter twice” predicts*d*, at similar program length, so no continuation is unquestionable from these five letters only\.

LLMs as universal Turing machines \(UTMs\)\.A single transformer forward pass is not Turing\-complete, but the autoregressive loop with an unbounded generated context is computationally universal\([Pérez et al\., 2021](https://arxiv.org/html/2610.14005#bib.bib59);[Schuurmans, 2023](https://arxiv.org/html/2610.14005#bib.bib60)\); the model’s chain of thought plays the role of a working tape\. Deterministic decoding can make an LLM work as a reference machine\. Also, the view of language models as compressors\([Deletang et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib32)\)suggests that the use of the logprobs for prompts is the right homologue to bits in a UTM\.

Saturation and the ARC challenges\.New benchmarks get saturated soon, because AI becomes more powerful, new models ‘benchmax’ or are contaminated, in a challenge\-solve\-and\-replace dynamics\([Zellers et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib104);[Schlangen, 2021](https://arxiv.org/html/2610.14005#bib.bib105)\)\. This is epitomised by Humanity’s Last Exam\([Center for AI Safety et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib77);[Phan et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib103)\), a compendium of very difficult problems in science that required the most expert humans for conception\. Other tests aim at what ‘is missing in AI’, such as the famous ARC \(Abstraction and Reasoning Challenge\) benchmark series\([ARC Prize Foundation, 2026](https://arxiv.org/html/2610.14005#bib.bib45)\)\. Originally conceived from algorithmic information theory\([Chollet, 2019](https://arxiv.org/html/2610.14005#bib.bib67)\), following[Hernández\-Orallo \(2017\)](https://arxiv.org/html/2610.14005#bib.bib58), the formalism, however, plays no operative role in the test\. Items have been mostly hand\-designed, difficulty is calibrated against human performance, and several versions of the challenge have saturated as usual, forcing a complete redesign\. Multiple ARC variants abound\([Maasch et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib44)\), now converging towards the old idea of evaluating AI with video games\([Pérez\-Liébana et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib36);[Ying et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib46);[Ying et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib47)\)\.

Well\-founded SuperARC\.To our knowledge, the C\-test remained the only test whose items were*generated and calibrated by*algorithmic information theory, together with its interactive variants\([Legg and Veness, 2013](https://arxiv.org/html/2610.14005#bib.bib72)\)\. Recently, SuperARC\([Hernández\-Espinosa et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib68)\)has revisited much of the original C\-test programme: sequence items of increasing algorithmic complexity\([Zenil et al\., 2018](https://arxiv.org/html/2610.14005#bib.bib75)\), and an accompanying thesis that compression is comprehension, recapitulating the motivation and name of the C\-test \(the C was for comprehension\)\. However, SuperARC is composed of a fixed inventory of \(mostly binary\) sequences, no unquestionability requirement is imposed, allowing for rival continuations\. Shared cultural knowledge is treated as contamination, to be removed by the retreat to binary strings\. Unsurprisingly, in this narrow context their central empirical finding is that when LLMs are asked for the pattern behind a sequence, they mostly return the equivalent of “print the sequence”\.

Verification, Scales and Scalability of AI Evaluation\.Creating tests with calibrated difficulty for future intelligence is a major challenge in AI evaluation\. The use of humans for judging the evaluation results does not scale up, with LLM\-as\-judges and other kinds of automated verification being commonplace\. However, there is a convoluted relationship between the verifier’s capabilities and item difficulty\([Zhou et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib53)\)\. Even if verification is clear in some domains \(e\.g\., in theorem proving\), the scales of difficulty are not known until a solution is found\. Also,[Achille and Soatto \(2026\)](https://arxiv.org/html/2610.14005#bib.bib70)have connected the capability of reasoning models with Levin search, but only for verifiable tasks\. Alternatively, populational measures of difficulty, such as IRT\([Martínez\-Plumed et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib87);[Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\), cannot anticipate the difficulty of an item that has not been solved yet\. But we can start with the solution of an inverse problem to create an item, and explore its properties to estimate difficulty, intrinsically\.

Prompt compression\.The greatest limitation of approaches based on algorithmic information theory is program size minimisation\. When this is applied to LLMs, the problem reappears\. Namely, finding the shortest prompt producing a target output is intractable in general; gradient\-guided discrete search\([Shin et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib61);[Zou et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib62);[Wen et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib63)\)and amortised language\-model inversion\([Morris et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib64)\)provide upper bounds\. Also, these techniques are not applicable for reasoning models, as the final part of the output is what matters\. Our pipeline avoids enumeration by using an LLM\-informed search of more compact prompt candidates\.

## 3LLM Costing and Metrics

We will use several LLMs for different roles\. Reference machines, used for costing, are frozen, versioned LLMs, using a deterministic decoding policy and tokens proceed autoregressively until a stopping condition, usually a designated delimiter⟨𝗈𝗎𝗍⟩\\langle\\mathsf\{out\}\\rangle\. This constitutes the model’s*chain of thought*, allowing nontrivial computations \(e\.g\., calendar arithmetic\) before emitting the final answer\. For simplicity, given a machine \(LLM\)ℳ\\mathcal\{M\}, a promptppand a contexttt, we denote byℳ⁡\(p\|t\)\\mathcal\{M\}\(p\|t\)the final output ofℳ\\mathcal\{M\}, andτℳ​\(p\|t\)\\tau\_\{\\mathcal\{M\}\}\(p\|t\)the number of tokens that have been used during the whole inference\. Different models have different vocabulariesΣℳ\\Sigma\_\{\\mathcal\{M\}\}and tokenisers\. For many open\-weight models we have access toPℳ​\(ti∣t1​…​ti−1\)P\_\{\\mathcal\{M\}\}\\\!\\left\(t\_\{i\}\\mid t\_\{1\}\\dots t\_\{i\-1\}\\right\), the probabilityℳ\\mathcal\{M\}assigns to tokentit\_\{i\}when preceded byt1:i−1t\_\{1:i\-1\}\. This is the prior the model represents, usually human language in a particular culture\.

Token count is a misleading program length: with\|Σℳ\|≈105\|\\Sigma\_\{\\mathcal\{M\}\}\|\\approx 10^\{5\}, every token would carrylog2⁡\|Σℳ\|≈17\\log\_\{2\}\|\\Sigma\_\{\\mathcal\{M\}\}\|\\approx 17bits, but tokens are not equally informative\. Instead, we use a different coding for all inputs and outputs:

###### Definition 3\.1\(Costing\)\.

The*cost*of any strings∈Σℳ∗s\\in\\Sigma^\{\*\}\_\{\\mathcal\{M\}\}conditioned to a contextt∈Σℳ∗t\\in\\Sigma^\{\*\}\_\{\\mathcal\{M\}\}, wheres=s1​…​sns=s\_\{1\}\\dots s\_\{n\}andt=t1​…​tmt=t\_\{1\}\\dots t\_\{m\}, is its Shannon–Fano code length under the model prior,

cℳ\(s\|t\)≔−∑i=1nlog2Pℳ\(si∣t1…tms1…si−1\)\.c\_\{\\mathcal\{M\}\}\(s\|t\)\\;\\coloneqq\{\}\\;\-\\sum\_\{i=1\}^\{n\}\\log\_\{2\}P\_\{\\mathcal\{M\}\}\\\!\\left\(s\_\{i\}\\mid t\_\{1\}\\dots t\_\{m\}s\_\{1\}\\dots s\_\{i\-1\}\\right\)\.

With no context, by Kraft’s inequality∑s2−cℳ​\(s\)≤1\\sum\_\{s\}2^\{\-c\_\{\\mathcal\{M\}\}\(s\)\}\\leq 1,cℳc\_\{\\mathcal\{M\}\}behaves as a prefix\-free program\-length measure\.

###### Definition 3\.2\(LLM\-relative Kolmogorov complexity\)\.

For any stringsssandtt,

Kℳ𝒱​\(s\|t\)≔minp⁡\{cℳ​\(p\|t\):𝒱⁡\(p\|t\)=s\},K\_\{\\mathcal\{M\}\}^\{\\mathcal\{V\}\}\(s\|t\)\\;\\coloneqq\\;\\min\_\{p\}\\left\\\{\\,c\_\{\\mathcal\{M\}\}\(p\|t\)\\;:\\;\\mathcal\{V\}\(p\|t\)=s\\,\\right\\\},where𝒱⁡\(p\|t\)\\mathcal\{V\}\(p\|t\)is the final output of𝒱\\mathcal\{V\}\. We assumessandttbelong to bothΣM∗\\Sigma^\{\*\}\_\{M\}andΣV∗\\Sigma^\{\*\}\_\{V\}, even if the tokenisations may be different, and equivalence is performed at the level of character encoding \(standard strings\)\.

This is a more general definition than the standard Kolmogorov complexity in that apart from checking that the program reproduces the input with a reference machine𝒱\\mathcal\{V\}, we have an additional reference machineℳ\\mathcal\{M\}to cost the program, conditional tott, instead of mere length\. When the superindex is not specified, we will assume both are the same machine\. Even in this case, the definition differs from standard Kolmogorov complexity, because of the use of a coding that depends ontt\. As we will see, if𝒱\\mathcal\{V\}naturally follows withsswhen giventt, as when an LLM reads a prompt “1,3,5,7,9” and computes “11, 13, \.\.\.”, thenp=ϵp=\\epsilon, the empty string, andKℳ𝒱​\(s\|t\)=0K\_\{\\mathcal\{M\}\}^\{\\mathcal\{V\}\}\(s\|t\)=0\.

###### Definition 3\.3\(LLM\-relative Levin complexity\)\.

For any stringsssandtt,

K​tℳ𝒱​\(s\|t\)≔minp⁡\{cℳ​\(p\|t\)\+log2⁡τ𝒱​\(p\|t\):𝒱⁡\(p\|t\)=s\},Kt\_\{\\mathcal\{M\}\}^\{\\mathcal\{V\}\}\(s\|t\)\\;\\coloneqq\\;\\min\_\{p\}\\left\\\{\\,c\_\{\\mathcal\{M\}\}\(p\|t\)\+\\log\_\{2\}\\tau\_\{\\mathcal\{V\}\}\(p\|t\)\\;:\\;\\mathcal\{V\}\(p\|t\)=s\\,\\right\\\},where𝒱⁡\(p\|t\)\\mathcal\{V\}\(p\|t\)is the final output of𝒱\\mathcal\{V\}, butτ𝒱​\(p\|t\)\\tau\_\{\\mathcal\{V\}\}\(p\|t\)is computed with all the inference tokens, including CoT\.

This is theconditionalLevin complexity ofssgiventt, and shortened byK​t\|Kt^\{\|\}\. The unit is bits, as the logarithm of the combinations of the original search perplexity2cℳ​\(p\|t\)​τ𝒱​\(p\|t\)2^\{c\_\{\\mathcal\{M\}\}\(p\|t\)\}\\tau\_\{\\mathcal\{V\}\}\(p\|t\)\. The use ofcℳc\_\{\\mathcal\{M\}\}rather than the length in tokens is justified because this search can be guided by the model probabilities \(see e\.g\.,[Achille and Soatto \(2026\)](https://arxiv.org/html/2610.14005#bib.bib70)\)\.

## 4Sequential Problems and Unquestionability

The above definitions are generic, and can work for arbitrary text sequences given as problem formulation, but also for inverse problems in the spirit of the original C\-test, and similar tests, as in[Thurstone \(1937\)](https://arxiv.org/html/2610.14005#bib.bib84)’s letter series\.

###### Definition 4\.1\(Sequential inversion problem\)\.

A sequential inversion problem is a comma\-separated text stringxxthat is composed of a prefixx1:nx\_\{1:n\}and a postfixxn\+1:n\+kx\_\{n\+1:n\+k\}\. Each of then\+kn\+kelements is known as a term, and can consist of one or more tokens\. The goal is to extrapolate the postfix having only seen the prefix\.

We will use the same subindex notation as before, now referring to terms, not tokens\. For instance,x1:nx\_\{1:n\}= “Italy, Slovenia, Croatia, Bosnia and Herzegovina, Montenegro” with prefix cut point atn=5n=5, includes terms that have more than one token\. Namely,x4x\_\{4\}isBosnia and Herzegovina\.

Now, is there a mechanism that unequivocally produces the postfix given the prefix? In this case, the pattern could bepp= “Mediterranean countries clockwise from Italy”\. First, we can figure out thatpp, when executed on𝒱\\mathcal\{V\}, will produce a string whose firstkkterms will match the postfix, i\.e\.,𝒱\(p\|x1:n\)=xn\+1:n\+k\\mathcal\{V\}\(p\|x\_\{1:n\}\)=x\_\{n\+1:n\+k\}\. This is the verification\. Also, we expectppto have lower cost than the listing programqqthat simply enumerates the following countries, but also lower than making a minor modification to programppsuch asp′=p^\{\\prime\}=“Mediterranean countries clockwise from Italy, except Croatia”\. Otherwise,p′p^\{\\prime\}could be a reasonable pattern forxx\.

While many philosophers of induction have argued that there is no reason to choose one mechanism over another\([Sterkenburg, 2016](https://arxiv.org/html/2610.14005#bib.bib41);[Dingle et al\., 2018](https://arxiv.org/html/2610.14005#bib.bib42)\), the original C\-test demanded*unquestionability*: among all programs ofnear\-minimal costthat reproduce the shown prefix, all must agree on the next term\. In our setting, formally\.

###### Definition 4\.2\(Unquestionability atLL\)\.

Letx=x1​x2​…x=x\_\{1\}x\_\{2\}\\dotsbe a comma\-separated sequence,x1:Lx\_\{1:L\}its firstLLterms, andxL\+1:L\+kx\_\{L\+1:L\+k\}the intended next terms\. Letp⋆p^\{\\star\}be a program generatingxxon𝒱\\mathcal\{V\}that minimises the unconditionalKℳV​\(x\)K\_\{\\mathcal\{M\}\}^\{V\}\(x\)=cℳ​\(p⋆\)c\_\{\\mathcal\{M\}\}\(p^\{\\star\}\)\. The itemxxis*unquestionable*atLLon𝒱\\mathcal\{V\}andℳ\\mathcal\{M\}if every programqqwithcℳ​\(q\)≤cℳ​\(p⋆\)\+δℳc\_\{\\mathcal\{M\}\}\(q\)\\;\\leq\\;c\_\{\\mathcal\{M\}\}\(p^\{\\star\}\)\+\\delta\_\{\\mathcal\{M\}\}and𝒱\(q\)1:L=x1:L\\mathcal\{V\}\(q\)\_\{1:L\}=x\_\{1:L\}also satisfies𝒱\(q\)L\+1:L\+k=xL\+1:L\+k\\mathcal\{V\}\(q\)\_\{L\+1:L\+k\}=x\_\{L\+1:L\+k\}\.

Typically, we look for the smallestLLsuch that unquestionability holds, defining the cutpointnnbetween prefix and postfix\. Complex patterns require a higherLL, for the terms to justify the program\. The toleranceδℳ\\delta\_\{\\mathcal\{M\}\}looks like a parameter but it is derived as the cost of the least expensive exception clause onℳ\\mathcal\{M\}, minimising on ‘except \.\.”, or “and fail randomly”, etc\. Accordingly, “p⋆p^\{\\star\}, butxL\+1′x^\{\\prime\}\_\{L\+1\}at termL\+1L\{\+\}1” is not a valid rival program\. Several behaviours are possible asLLgrows: a comprehensive program \(“the odd numbers”\) has cost*constant*inLL; a program that patches a rule with exceptions pinned to positions \(“the odd numbers except number 7 and 19”\) has a cost that grows bylog⁡L\\log Lif more exceptions are needed along the way; alistingprogrampλp^\{\\lambda\}that just enumerates the terms \(“printx1​…​xL,xL\+1x\_\{1\}\\dots x\_\{L\},x\_\{L\+1\}”\) has cost growing aroundβ​L\\beta L, whereβ\\betais the mean per\-term cost\. In general, showing more terms \(i\.e\., increasingLL\) eventually puts all rivals out of theδℳ\\delta\_\{\\mathcal\{M\}\}\-window\.

As a listing will work for any finite continuation,[Definition4\.2](https://arxiv.org/html/2610.14005#S4.Thmdefinition2)implies that the program must compressxx:

###### Proposition 4\.1\(Compressibility\)\.

If an itemxxis unquestionable atLL, thenc⁡\(p⋆\)≤c⁡\(pλ\)−δc\(p^\{\\star\}\)\\leq c\(p^\{\\lambda\}\)\-\\delta\. In words, the program with smallest coding must be simpler thanpλp^\{\\lambda\}\(the listing that enumerates the continuation\) by a marginδ\\delta\.

###### Corollary 4\.1\.

Assumec⁡\(pλ\)=β​L\+c0c\(p^\{\\lambda\}\)=\\beta L\+c\_\{0\}, withβ\\betabeing the per\-term cost ofpλp^\{\\lambda\}andc0c\_\{0\}a printing scaffold\. If the best programp⋆p^\{\\star\}for an itemxxhasc⁡\(p⋆\)\>β​L−δ\+c0c\(p^\{\\star\}\)\>\\beta L\-\\delta\+c\_\{0\}thenxxis questionable atLL\. \[Proof in Appendix[Q](https://arxiv.org/html/2610.14005#A17)\.\]

For minimal Turing machines, such as those in the original C\-test, loop or recursion are the standard sources for these sequentialtransformations, such as “xi\+1=xi\+2x\_\{i\+1\}=x\_\{i\}\+2loop forever”\. However, for an LLM, a good sequence can be obtained by retrieval only, such aspp= “the kings and queens of England”\. Note that this is not a listing, but a namedretrievalthat recovers a memorised listing\. The cost of thisppon a modern LLM is small \(a few common words\), and the probability that any decent LLM can recover the list is high\. This is perfectly okay: continuing a sequence such as “William I, William II, Henry I” is easy, and one of the motivations for using LLMs in the first place\. Many mechanisms are composed of zero or more transformations and zero or more retrievals\. Of course there will be differences between reference machines, and some of them will have better retrieval than transformation capacity\. Reasoning models will usually add to the latter\.

Retrieval has a cost\. For instance, indexing the On\-Line Encyclopedia of Integer Sequences \(OEIS\) requires almost 20 bits to name one\. Similarly to name a published book\. Transformations also pay when they are entropic, such as “3\.253​n0\.7343\.253n^\{0\.734\}”\. Actually, it is in the combination of retrieval and transformation where many patterns lie, such as “the weekdays backwards, interleaved with the months”\. As we said in the introduction, retrieval and transformation connect with the psychometric literature through crystallised and fluid intelligence, respectively\. These two dimensionsr,tr,tare simply used in the prompts for the composer, when populating the test battery\. However, it is unclear how the levels of retrieval and transformation will influence observed performance\.

## 5C\-test ONE: Pipeline

Borrowing from the original C\-test, we call the new test C\-test ONE\. Beyond the use of LLMs as UTMs for the representation of programs and their coding, the test has many other innovations, such as the use of domains and the retrieval\-transformation matrix in the generator, the intervention of a composer for unquestionability and, most especially, the use of the conditional version of Levin’s complexity,K​t\|Kt^\{\|\}\([Definition3\.3](https://arxiv.org/html/2610.14005#S3.Thmdefinition3)\), when certifying the items\.

1generate2validate3unquest\.4certify5evaluatedomainsr0r^\{0\},t0t^\{0\}p0p^\{0\}xxn=Lun=L\_\{u\}K^\\widehat\{K\},p▼p^\{\\blacktriangledown\}K^\|\\widehat\{K\}^\{\|\},τ^\\hat\{\\tau\}K​t^\|\\widehat\{Kt\}^\{\|\}ℰ\(x1:n\)=𝒥​?\\mathcal\{E\}\(x\_\{1:n\}\)\\overset\{\\mathcal\{J\}?\}\{=\}xn\+1:n\+kx\_\{n\+1:n\+k\}𝒞\\mathcal\{C\}𝒱\\mathcal\{V\}𝒞,𝒱,ℳ\\mathcal\{C\},\\mathcal\{V\},\\mathcal\{M\}ℳ\\mathcal\{M\}ℰ,𝒥\\mathcal\{E\},\\mathcal\{J\}an inverse problemexpressed in wordsit reproducesxxat least 2 out of 3no rival withinK^\+δ\\widehat\{K\}\+\\deltaatLuL\_\{u\}ℳ\\mathcal\{M\}can runa descriptionx1:nx\_\{1:n\}shown,xn\+1:n\+kx\_\{n\+1:n\+k\}askedFigure 1:C\-test ONE pipeline: items are generated followed by several filters, applied sequentially\. The items at the end are those that arevalidated,unquestionableandcertified\. Stages 1 and 2 are independent of the reference machineℳ\\mathcal\{M\}, but stages 3 and 4 apply filters that depend onℳ\\mathcal\{M\}\. Notation is explained in the text\.[Figure1](https://arxiv.org/html/2610.14005#S5.F1)shows the C\-test ONE pipeline, with five major stages\. \(1\) Generation: the composer works with a diverse pool of domains to integrate a number of retrieval \(r0r^\{0\}\) and transformation \(t0t^\{0\}\) operations\. The composer𝒞\\mathcal\{C\}generates fresh sequence mechanisms, called seeds,p0p^\{0\}, as text in natural language \(e\.g\., “The first letter of the chemical symbol of each atomic element”\)\. \(2\) Validation: for each seed a verifier𝒱\\mathcal\{V\}runs them as a standalone prompt, and checks the level of consistency\[𝒱\(p0\)=x\]2​@​3\[\\mathcal\{V\}\(p^\{0\}\)=x\]^\{2@3\}twice for a sufficient total sizeNN\. For instance, do we repeatedly getxx= “H, H, L, B, B, C, N, O, F, N, N, \.\.\.”? \(3\) Unquestionability forℳ\\mathcal\{M\}: Items are split between the prefix \(the observable part to the examinee,x1:nx\_\{1:n\}\) and the postfix \(what the examinee must guess,xn\+1:n\+kx\_\{n\+1:n\+k\}\)\. The point of splitLuL\_\{u\}is increased gradually until unquestionability holds\. The seed is used for the composer to propose more compact unconditional representations\{p:𝒱⁡\(p\)=x\}\\\{p:\\mathcal\{V\}\(p\)=x\\\}, referred to as ‘equivalents’, such as “enumerate first letter of elements”, and looks for unconditional mechanisms that reproduce the prefix but diverge on the postfix\{q:𝒱\(q\)1:n=x1:n,𝒱\(q\)n\+1:n\+k≠xn\+1:n\+k\}\\\{q:\\mathcal\{V\}\(q\)\_\{1:n\}=x\_\{1:n\},\\mathcal\{V\}\(q\)\_\{n\+1:n\+k\}\\neq x\_\{n\+1:n\+k\}\\\}, known as ‘rivals’, such as “Loop H, H, L, B, B, C forever” diverging fromn=7n=7\. The equivalent with lowest cost,p▼p^\{\\blacktriangledown\}, is used as a proxy of complexityK^ℳ𝒱​\(x\)=cM​\(p▼\)\\hat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=c\_\{M\}\(p^\{\\blacktriangledown\}\)\. If there is no rivalqqof a similar costcℳ​\(q\)c\_\{\\mathcal\{M\}\}\(q\)tocℳ​\(p▼\)c\_\{\\mathcal\{M\}\}\(p^\{\\blacktriangledown\}\)then the item is unquestionable atnn\. \(4\) Conditional certification forℳ\\mathcal\{M\}\. Conditional hints, such as “chemical elements” \(together with the seed, equivalents, etc\.\) are explored and checked onℳ\\mathcal\{M\}, i\.e\.,\{p:ℳ\(p\|x1:n\)=xn\+1:n\+k\}\\\{p:\\mathcal\{M\}\(p\|x\_\{1:n\}\)=x\_\{n\+1:n\+k\}\\\}, and metrics such asK​t\|Kt^\{\|\}are derived by minimising the conditional cost and then adding the logarithm of the number of inference tokens\. Finally, \(5\) Evaluation applies the battery to any examineeℰ\\mathcal\{E\}\. Difficulty and performance can be compared to draw characteristic curves\.

Asking the composer𝒞\\mathcal\{C\}, an LLM, to generate a diversity of items would not work by repeating the same prompt, with or without temperature\. To ensure some diversity, the generator works with a set of domains,themesthat are created and curated by the composer, such as “chess openings” and “Beaufort scale’\. The database grows as more domains are required, and the composer always keeps track of the ones already created to avoid repetition\. While we could inject domains manually, for the current battery everything has been generated automatically\.

To ensure diversity in retrieval and transformation levels in the item, the generator aims at populating a matrix, ranging betweenr=0\.\.3r=0\.\.3andt=0\.\.7t=0\.\.7\. The chosenr0r^\{0\}andt0t^\{0\}are sent to the composer to create a seed with that number of components\. However, because the composer can miscalculate or the seedsp0p^\{0\}may have retrievals and transformations that cancel, the actual values ofrrandttmay be different\. Because of this, the generator looks actively for gaps and interpolates between demanded values and estimated values, correcting this with a linear function\. As we can see in the\(r,t\)\(r,t\)matrices in the appendix, the procedure manages to fill almost every cell\.

[Table1](https://arxiv.org/html/2610.14005#S5.T1)gives a formal expression of the filters applied to each candidate seed\. For more details about the pipeline and the prompts used for the composer, we refer to Appendices[H](https://arxiv.org/html/2610.14005#A8)and[I](https://arxiv.org/html/2610.14005#A9)\.

Table 1:The three filters that are applied after generation\. The final row shows how evaluation works: an examineeℰ\\mathcal\{E\}is shownx1:Lux\_\{1:L\_\{u\}\}and asked forkf=5k\_\{f\}=5more terms\. All terms must be equal\.
## 6Experimental design

Experiments must be sufficient to answer several research questions\. RQ1: Can the pipeline create a battery of inverse sequences that are valid, unquestionable and certifiable using small LLMs? RQ2: Are they diverse in retrieval and transformation components and difficulty? RQ3: DoesK​t\|Kt^\{\|\}work as a calibrated measure of difficulty for both weak and strong models? RQ4: Is the ability obtained from the characteristic curves usingK​t\|Kt^\{\|\}aligned with populational ability metrics? RQ5: What is the supraintelligence window that we can obtain with this procedure, as a sign of its scalability? and RQ6: How does the power of the reference machine influence all these questions?

As a self\-imposed constraint for sustainability and reproducibility, the experimental design should not require more than the necessary resources to answer the research questions\. Accordingly, we estimated that for RQ1, RQ2, RQ4 and RQ5 we had to set a target of a minimum of 150 generated items with at least 30 being certified for each and every reference machine\. For RQ1, RQ3, RQ5 and RQ6, we chose three reference machines that can fit in a T4 GPU\. For RQ3, RQ4 and RQ5 we used 12 LLMs of different families, including commercial and open\-weight\. Also, for the logic in the pipeline we chose several Google Gemini variants because of their balance of speed, quality and price\. All the choices are shown in[Table2](https://arxiv.org/html/2610.14005#S6.T2)\. Three Anthropic models as examinees had high refusal percentages \(see[Table5](https://arxiv.org/html/2610.14005#A1.T5)in the appendix\), and we used versions with fallback, represented with \*\.

Table 2:LLMs used\. Models with a star used fallback, to reduce refusal percentages\. Temperature 0 unless stated\.The implementation of[Definition3\.2](https://arxiv.org/html/2610.14005#S3.Thmdefinition2)and[Definition3\.3](https://arxiv.org/html/2610.14005#S3.Thmdefinition3)is instantiated for sequences as follows:

Kℳ𝒱\(x1:n\+k\)≔minp\{cℳ\(p\):𝒱\(p\)=x1:n\+k\},K\_\{\\mathcal\{M\}\}^\{\\mathcal\{V\}\}\(x\_\{1:n\+k\}\)\\;\\coloneqq\\;\\min\_\{p\}\\left\\\{\\,c\_\{\\mathcal\{M\}\}\(p\)\\;:\\;\\mathcal\{V\}\(p\)=x\_\{1:n\+k\}\\,\\right\\\},Ktℳ\(xn\+1:n\+k\|x1:n\)≔minp\{cℳ\(p\|x1:n\)\+log2τℳ\(p,x1:n\):ℳ\(⟨p,x1:n⟩\)=xn\+1:n\+k\},Kt\_\{\\mathcal\{M\}\}\(x\_\{n\+1:n\+k\}\|x\_\{1:n\}\)\\;\\coloneqq\\;\\min\_\{p\}\\left\\\{\\,c\_\{\\mathcal\{M\}\}\(p\|x\_\{1:n\}\)\+\\log\_\{2\}\\tau\_\{\\mathcal\{M\}\}\(p,x\_\{1:n\}\)\\;:\\;\\mathcal\{M\}\(\\langle p,x\_\{1:n\}\\rangle\)=x\_\{n\+1:n\+k\}\\,\\right\\\},We have a specific prompt that is standard for all verifications \(see[Figure14](https://arxiv.org/html/2610.14005#A9.F14)in the appendix\), instead of a simple concatenation⟨p,x1:n⟩\\langle p,x\_\{1:n\}\\rangle\. Finally, we used many other baselines for comparisons, such as

Kcℳ\(xn\+1:n\+k\|x1:n\)≔minp\{cℳ\(p∣x1:n\):ℳ\(⟨p,x1:n⟩\)=xn\+1:n\+k\}\+cℳ\(x1:n\)Kc\_\{\\mathcal\{M\}\}\(x\_\{n\+1:n\+k\}\|x\_\{1:n\}\)\\;\\coloneqq\\;\\min\_\{p\}\\big\\\{\\,c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:n\}\)\\;:\\;\\mathcal\{M\}\(\\langle p,x\_\{1:n\}\\rangle\)=x\_\{n\+1:n\+k\}\\,\\big\\\}\\;\+\\;c\_\{\\mathcal\{M\}\}\(x\_\{1:n\}\)which adds a term measuring how unusual the prefix is, instead of the time to run the program\. We calculatedrrandtt\(original and estimated\) and the length in charactersℓc​h​a​r\\ell\_\{char\}of the programsp0p^\{0\}\(the seed\),p▽p^\{\\triangledown\}\(a bare version of the seed\),p▼p^\{\\blacktriangledown\}\(the unconditional program with lowest cost\), andp▼¯p^\{\\overline\{\\blacktriangledown\}\}\(the conditional program with lowest cost\)\.

## 7Results

We will include a selection of plots and tables, mostly for Qwen3\-14B, since it is the reference machine that certified the highest number of items in the battery\. More plots for Qwen3\-14B can be found in Appendix[A](https://arxiv.org/html/2610.14005#A1), and all the results for Qwen3\-8B and Phi\-4\-14B\-reasoning can be found in appendices[B](https://arxiv.org/html/2610.14005#A2)and[C](https://arxiv.org/html/2610.14005#A3)respectively\.

The full battery had 151 items, of which 107 were valid\. From those, Qwen3\-14B made 71 unquestionable, of which 51 were certified\. Qwen3\-8B increased unquestionable items to 76 but certified only 32, due to its incapability of successfully running any hint or program for complex items given the budget\. Finally, Phi\-4\-reasoning is placed in between, with 80 unquestionable and 38 certified\.RQ1 is answered affirmatively\.

For Qwen3\-14B specifically,[Figure2](https://arxiv.org/html/2610.14005#S7.F2)shows the percentage of items being solved by each examinee for each cell in the grid\.[Table7](https://arxiv.org/html/2610.14005#A1.T7)in the appendix gives the full breakdown more explicitly\. Highttwith lowrrseems to be a specific area where models tend to fail\.RQ2 is answered affirmatively\.

![Refer to caption](https://arxiv.org/html/2610.14005v1/fig_heat_unq_fine_full_qwen3-14b_bare.png)Figure 2:The 12 LLM examinees on the \(r^,t^\\hat\{r\},\\hat\{t\}\) grid and accuracy for the unquestionable items for Qwen3\-14B\.[Table3](https://arxiv.org/html/2610.14005#S7.T3)shows Spearman correlations for a range of metrics per model for the 51 certified items under Qwen3\-14B\.K​t\|Kt^\{\|\}works very well \(within sampling error withK​c\|Kc^\{\|\}andK\|K^\{\|\}, see Appendix[F](https://arxiv.org/html/2610.14005#A6)\) with strong correlations per model \(except the weak examinees\) and globally\.RQ3 is answered affirmatively\.

Table 3:Spearman correlations between several metrics and full correct response for the 51 valid, unquestionable and certified items under Qwen3\-14B\. Each item had three repetitions per examinee\. All metrics in[Table8](https://arxiv.org/html/2610.14005#A1.T8)\.[Figure3](https://arxiv.org/html/2610.14005#S7.F3)\(left\) shows the logistic fit of the points for the 51 certified examples for Qwen3\-14B \(2​N2Nprior points added at \(0,1\) as anchor\)\. Empirical curves in the appendix \([Figure5](https://arxiv.org/html/2610.14005#A1.F5)\)\. The plot on the right compares the ability of each model with Item Response Theory estimates of the abilities of the same models according to Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\. Correlation is high, despite the different scales\.RQ4 is answered affirmatively\.

Figure 3:Left: Agent characteristic curves for the 12 LLM, withK​t\|Kt^\{\|\}provided by Qwen3\-14B \(51 certified examples\), with a logistic fit, where the ability of each modelθ\\theta, measured in bits, is estimated by the point where performance is 0\.5\. Right: Comparison between each model’sabilityand the psychometric, populational score given by Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\. Only 10 of our 12 are listed in ECI\.[Table4](https://arxiv.org/html/2610.14005#S7.T4)shows the infra and supra\-intelligence windows for the three reference models\. The infra window runs from the item with lowestK​t\|Kt^\{\|\}to the point where the fitted logistic curve crosses 2/3 accuracy\. Similarly, the supra window runs from where it falls through 1/3 to the item with highestK​t\|Kt^\{\|\}\. While the interval between the windows is where information is high, these windows show the margin for discrimination in both ends of the evaluation spectrum\. Qwen3\-8B does not have enough resolution in the higher end of the scale\. Qwen3\-14B has limited sensitivity in the other end, but its upper end shows the scalability of the construction\. Even for a model such as gpt5sol, a much weaker model such as Qwen3\-14B can generate a very ample, yet sparse window between 190\-380 where the fitted curve falls below 1/3 \(Appendix[F](https://arxiv.org/html/2610.14005#A6)\)\.RQ5 is answered by the supraintelligence window being bounded and sparse, yet potentially accommodating models that are much stronger than the reference machine\.

Table 4:Infra\- and supra\-intelligence windows of all examinees per reference machine\. The supra ranges for Qwen3\-14B are wide, indicating it can potentially challenge frontier models\. Ill\-defined windows in grey\.[Figure4](https://arxiv.org/html/2610.14005#S7.F4)shows how theK​t\|Kt^\{\|\}metric changes between two reference machines of the same family\. Not only are many items not certified by Qwen3\-8B, as compared to Qwen3\-14B, but their difficulty is higher\. Qwen3\-8B is a worse compressor, but mostly because many equivalent programs and hints that reduceK​t\|Kt^\{\|\}in Qwen3\-14B do not work for Qwen3\-8B\. Accordingly,RQ6 is partially answered positivelyby this plot, but the appendix contains the corresponding plots and tables for all the reference machines, to observe more precisely how they affect the generation and results\. The other two pairwise combinations can be found in[Figure8](https://arxiv.org/html/2610.14005#A2.F8)and[Figure12](https://arxiv.org/html/2610.14005#A3.F12)in the appendix\.

Figure 4:Comparison ofK​t\|Kt^\{\|\}between Qwen\-3\-8B and Qwen\-3\-14B vs average results of 12 other LLMs on the 71 unquestionable items for Qwen\-3\-14B\. Success decreases withK​t\|Kt^\{\|\}\. The weaker model \(Qwen\-3\-8B, purple\) brings more refinement for lowK​t\|Kt^\{\|\}\(left\), but struggles to cost many items of highK​t\|Kt^\{\|\}\(right\)\. An arrow pointing left means Qwen\-3\-14B describes the item at less cost\. Values near zero are empty programs, withK​t=0\+log2⁡τKt=0\+\\log\_\{2\}\\tau\.
## 8Limitations and Future Work

The choice of very small LLMs as reference machines and a commodity model as composer shows the pipeline works with low resources\. Conducting low\-budget evaluations that make frontier models fail predictably and gradually gives a positive signal towards the scalability of evaluations\. So far, the use of Kolmogorov complexity has usually been objected to because of its incomputability and subjectivity\. We use Levin’sK​tKt, which is computable, as it considers the number of tokens in the reasoning steps in a refurbished universal search expression that turns out to be fundamental\([Achille and Soatto, 2026](https://arxiv.org/html/2610.14005#bib.bib70)\)\. Subjectivity toℳ\\mathcal\{M\}is partially addressed in terms of ratio scales properties in Appendix[D](https://arxiv.org/html/2610.14005#A4)\. Of course, given the infinite options forℳ\\mathcal\{M\}, an explicit choice as an LLM makes the complexity measure both meaningful for humans and also for the current paradigm in AI\.

One of the limitations comes precisely from the bias on human language and culture: the generator produces text in English, dominated by Western culture\. This is a convenience, but the battery and costing can be parametrised to other languages and cultures by changing𝒞\\mathcal\{C\}andℳ\\mathcal\{M\}\. A second limitation is that C\-test ONE is based only on sequences\. The purpose of this paper was to show the approach works\. The next step could be non\-sequential problems, such as agential scenarios \(e\.g\., mini\-games,\([Ying et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib46);[Ying et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib47)\)\) where the solution of the game \(not its description as in\([Legg and Hutter, 2007](https://arxiv.org/html/2610.14005#bib.bib71)\)\) is quantified according toK​t​\(p\|x\)Kt\(p\|x\)\. But note that applying the processes of minimisation and unquestionability a posteriori, may expose the weakness of existing batteries such as\([ARC Prize Foundation, 2026](https://arxiv.org/html/2610.14005#bib.bib45)\)\. Instead, generating items adapting our procedure can create more robust tests\. Nonetheless, the possibility of calculating intrinsic properties of existing benchmarks, such as their difficulty, and their fluid and crystallised components, should be at the core of a staircase of batteries we should use for frontier AI\.

As future work, the generator can be improved to speed up the estimates ofK​t\|Kt^\{\|\}\([Orseau et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib80);[Li, 2026](https://arxiv.org/html/2610.14005#bib.bib78);[Tuero et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib79)\), and more reference machines, including multimodal, could always be explored, and combined\. Beyond these optimisations, the potential of this paper lies in rebuilding the foundations of AI evaluation broadly\.

The lesson learnt is that with modest budget and models we can generate a systematic class of verifiable questions beyond current frontier capabilities, with their difficulty measured in a theoretically\-grounded way, populating a supra\-intelligence window that extends our potential for evaluating the frontier\. C\-test ONE is the first in a series achieving this\. Let others climb the next steps in the ladder\.

### AI use statement

We used so\-called generative AI in several stages of this project, other than the test itself \(with LLMs being used as verifiers, composers or reference machines\)\. Several LLMs were used for idea generation and brain\-storming by the authors, and helped with locating and processing related work, but all papers and books that are cited were read, or at least scanned for the relevant passages, by the authors\. References were checked manually with standard sources \(e\.g\., google scholar\), which may still contain errors\. LLMs were used extensively for vibe coding, processing the results, and generating tables, plots and other result analyses\. The logs and results were the main way in which the code was verified to work correctly\. LLMs were also used to draft early versions of some sections, especially those associated with the experiments, but all text was finally rewritten by the authors\. Some illustrations were also partly generated by AI\. We take full responsibility for all the content and associated code and datasets, especially the parts that were drafted by AI\.

### Ethics statement

This work does not have any special ethical issues other than the general concerns of a crucial field such as AI evaluation\. Problems in underestimation or overestimation of capabilities may have an impact on the appropriate deployment of AI, including areas such as fairness and safety\. The focus of this paper is put on the scalability of evaluations, and some level of delegation of this evaluation, which may entail some risks\. Another element of concern is the implications of this research for the environment\. We have devised the experimental setting in such a way that we performed the experiments that were sufficient to answer our research questions and support any claim that we made\. We also report a detailed and comprehensive use of compute and its environmental impact in Appendix[O](https://arxiv.org/html/2610.14005#A15)\. On purpose, we did not include very large reference machines or a very wide battery of API models to keep our impact as low as possible\. We think the variations, ablations, repetitions and numbers of models and items that we are using are reasonable\. Apart from this, we did not conduct any experiment with humans or animals, or included data from these sources directly \(although we used API calls, and we cannot be sure of how proprietary models were trained\)\.

### Reproducibility statement

Appendices[H](https://arxiv.org/html/2610.14005#A8)to[K](https://arxiv.org/html/2610.14005#A11)include a detailed description of the pipeline, the prompts used, the description of all notebooks and data files in the project, as well as all the parameters we used in the experiments\. We also include the full battery in appendix[P](https://arxiv.org/html/2610.14005#A16)\. All code and data are available at[https://github\.com/jorallo/ctest\-one](https://github.com/jorallo/ctest-one)\. Also, despite our self\-imposed limitation on the number of GPU/CPU hours and LLM tokens, our detailed report of the use of compute and tokens in appendix[O](https://arxiv.org/html/2610.14005#A15)helps other researchers anticipate their costs of reproduction\.

#### Acknowledgments

I’m thankful to Jan Arne Telle for insightful comments on a very early version of this project\.

This paper would have not been possible without the help of the other co\-author of the original C\-test paper, Neus Minaya\-Collado, who this time helped with extra ‘compute’ for this project\. The time was ripe after thirty years\.

## References

- M\. Abdin, S\. Agarwal, A\. Awadallah, V\. Balachandran, H\. Behl, L\. Chen, G\. de Rosa, S\. Gunasekar, M\. Javaheripi, N\. Joshi, P\. Kauffmann, Y\. Lara, C\. C\. T\. Mendes, A\. Mitra, B\. Nushi, D\. Papailiopoulos, O\. Saarikivi, S\. Shah, V\. Shrivastava, V\. Vineet, Y\. Wu, S\. Yousefi, and G\. ZhengPhi\-4\-reasoning technical report\.arXiv preprint arXiv:2504\.21318\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Achille and Soatto \(2026\)A\. Achille and S\. SoattoAI agents as universal task solvers\.Entropy28\(3\),pp\. 332\.Cited by:[Appendix F](https://arxiv.org/html/2610.14005#A6.p5.1),[§2](https://arxiv.org/html/2610.14005#S2.p7.1),[§3](https://arxiv.org/html/2610.14005#S3.p5.1),[§8](https://arxiv.org/html/2610.14005#S8.p1.1)\.
- Anthropic \(2025\)AnthropicClaude Haiku 4\.5 system card\.Note:[https://www\.anthropic\.com/news/claude\-haiku\-4\-5](https://www.anthropic.com/news/claude-haiku-4-5)Released with the modelCited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Anthropic \(2026a\)AnthropicClaude Sonnet 5 system card\.Note:[https://www\.anthropic\.com/claude\-sonnet\-5\-system\-card](https://www.anthropic.com/claude-sonnet-5-system-card)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Anthropic \(2026b\)AnthropicSystem card: Claude Fable 5\.1 & Claude Mythos 5\.1\.Note:[https://www\-cdn\.anthropic\.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205\.1%20&%20Claude%20Mythos%205\.1%20System%20Card\.pdf](https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Anthropic \(2026c\)AnthropicSystem card: Claude Opus 5\.Note:[https://www\-cdn\.anthropic\.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card\.pdf](https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- ARC Prize Foundation \(2026\)ARC Prize FoundationARC\-AGI\-3: a new challenge for frontier agentic intelligence\.arXiv preprint arXiv:2603\.24621\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p5.1),[§8](https://arxiv.org/html/2610.14005#S8.p2.1)\.
- Balloccuet al\.\(2024\)S\. Balloccu, P\. Schmidtová, M\. Lango, and O\. DušekLeak, cheat, repeat: data contamination and evaluation malpractices in closed\-source llms\.arXiv preprint arXiv:2402\.03927\.Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1)\.
- Bock and Gibbons \(2021\)R\. D\. Bock and R\. D\. GibbonsItem response theory\.John Wiley & Sons\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Cattell \(1963\)R\. B\. CattellTheory of fluid and crystallized intelligence: a critical experiment\.Journal of Educational Psychology54\(1\),pp\. 1–22\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Center for AI Safetyet al\.\(2026\)Center for AI Safety, Scale AI, and HLE Contributors ConsortiumA benchmark of expert\-level academic questions to assess AI capabilities\.Nature649\(8099\),pp\. 1139–1146\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09962-4)Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Chollet \(2019\)F\. CholletOn the measure of intelligence\.arXiv:1911\.01547\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Cobbeet al\.\(2020\)K\. Cobbe, C\. Hesse, J\. Hilton, and J\. SchulmanLeveraging procedural generation to benchmark reinforcement learning\.InInternational conference on machine learning,pp\. 2048–2056\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Cook \(1971\)S\. A\. CookThe complexity of theorem proving procedures\.InProceedings of the Third Annual ACM Symposium on Theory of Computing,pp\. 151–158\.External Links:[Document](https://dx.doi.org/10.1145/800157.805047)Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p1.1)\.
- Deletanget al\.\(2024\)G\. Deletang, A\. Ruoss, P\. Duquenne, E\. Catt, T\. Genewein, C\. Mattern, J\. Grau\-Moya, L\. K\. Wenliang, M\. Aitchison, L\. Orseau, M\. Hutter, and J\. VenessLanguage modeling is compression\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jznbgiynus)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p4.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Dingleet al\.\(2018\)K\. Dingle, C\. Q\. Camargo, and A\. A\. LouisInput–output maps are strongly biased towards simple outputs\.Nature communications9\(1\),pp\. 761\.Cited by:[§4](https://arxiv.org/html/2610.14005#S4.p4.1)\.
- Douet al\.\(2025\)H\. Dou, Z\. Li, J\. Du, L\. Yang, W\. Yao, and Y\. DengHybrid regularization improves diffusion\-based inverse problem solving\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7pr2doXn3)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1)\.
- Dowe and Hájek \(1997\)D\. L\. Dowe and A\. R\. HájekA computational extension to the Turing test\.InProceedings of the 4th Conference of the Australasian Cognitive Science Society,Newcastle, NSW, Australia\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Embretson and Reise \(2013\)S\. E\. Embretson and S\. P\. ReiseItem response theory\.Psychology Press\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Epoch AI \(2025\)Epoch AIEpoch Capabilities Index \(ECI\)\.Note:[https://epoch\.ai/eci](https://epoch.ai/eci)Data under CC BY 4\.0, accessed September 2026Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Freund \(2019\)R\. FreundRasch and rationality: scale typologies as applied to item response theory\.Ph\.D\. Thesis,UC Berkeley\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Gemma Team \(2025\)Gemma TeamGemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3\.1 Flash\-Lite model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-flash\-lite/](https://deepmind.google/models/model-cards/gemini-3-1-flash-lite/)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.5 Flash\-Lite model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-5\-flash\-lite/](https://deepmind.google/models/model-cards/gemini-3-5-flash-lite/)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Google DeepMind \(2026c\)Google DeepMindGemini 3\.6 Flash model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-6\-flash/](https://deepmind.google/models/model-cards/gemini-3-6-flash/)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Google DeepMind \(2026d\)Google DeepMindGemini 3\.8 Flash model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-8\-flash/](https://deepmind.google/models/model-cards/gemini-3-8-flash/)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Groetsch \(1999\)C\. W\. GroetschInverse problems: activities for undergraduates\.Vol\.12,Cambridge University Press\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Guggeret al\.\(2022\)S\. Gugger, L\. Debut, T\. Wolf, P\. Schmid, Z\. Mueller, S\. Mangrulkar, M\. Sun, and B\. BossanAccelerate: training and inference at scale made simple, efficient and adaptable\.Note:[https://github\.com/huggingface/accelerate](https://github.com/huggingface/accelerate)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Hand \(2016\)D\. J\. HandMeasurement: a very short introduction\.Oxford University Press\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Harriset al\.\(2020\)C\. R\. Harris, K\. J\. Millman, S\. J\. van der Walt, R\. Gommers, P\. Virtanen, D\. Cournapeau, E\. Wieser, J\. Taylor, S\. Berg, N\. J\. Smith, R\. Kern, M\. Picus, S\. Hoyer, M\. H\. van Kerkwijk, M\. Brett, A\. Haldane, J\. Fernández del Río, M\. Wiebe, P\. Peterson, P\. Gérard\-Marchant, K\. Sheppard, T\. Reddy, W\. Weckesser, H\. Abbasi, C\. Gohlke, and T\. E\. OliphantArray programming with NumPy\.Nature585\(7825\),pp\. 357–362\.External Links:[Document](https://dx.doi.org/10.1038/s41586-020-2649-2)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Hernández\-Espinosaet al\.\(2026\)A\. Hernández\-Espinosa, L\. Ozelim, F\. S\. Abrahão, and H\. ZenilSuperARC: a test for artificial superintelligence based on compressed modelling, recursive prediction and problem complexity\.Nature Communications17,pp\. 4885\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p6.1)\.
- Hernández\-Orallo and Minaya\-Collado \(1998\)J\. Hernández\-Orallo and N\. Minaya\-ColladoA formal definition of intelligence based on an intensional variant of algorithmic complexity\.InProceedings of the International Symposium of Engineering of Intelligent Systems \(EIS’98\),pp\. 146–163\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p3.1)\.
- Hernández\-Orallo \(2000\)J\. Hernández\-OralloBeyond the Turing test\.Journal of Logic, Language and Information9\(4\),pp\. 447–466\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p3.1)\.
- Hernández\-Orallo \(2015\)J\. Hernández\-OralloC\-tests revisited: back and forth with complexity\.InArtificial General Intelligence \(AGI 2015\),LNCS, Vol\.9205,pp\. 272–282\.Note:Extended version: arXiv:1412\.8529Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px2.p1.1)\.
- Hernández\-Orallo \(2017\)J\. Hernández\-OralloThe measure of all minds: evaluating natural and artificial intelligence\.Cambridge University Press\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px2.p1.1),[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Hoet al\.\(2025\)A\. Ho, J\. Denain, D\. Atanasov, S\. Albanie, and R\. ShahA Rosetta stone for AI benchmarks\.arXiv preprint arXiv:2512\.00193\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1),[Figure 7](https://arxiv.org/html/2610.14005#A2.F7),[Appendix B](https://arxiv.org/html/2610.14005#A2.p3.1),[Figure 11](https://arxiv.org/html/2610.14005#A3.F11),[Appendix C](https://arxiv.org/html/2610.14005#A3.p3.1),[§2](https://arxiv.org/html/2610.14005#S2.p7.1),[Figure 3](https://arxiv.org/html/2610.14005#S7.F3),[§7](https://arxiv.org/html/2610.14005#S7.p5.1)\.
- Huet al\.\(2025\)L\. Hu, Q\. Li, A\. Xie, N\. Jiang, I\. Stoica, H\. Jin, and H\. ZhangGameArena: evaluating LLM reasoning through live computer games\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SeQ8l8xo1r)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Hunter \(2007\)J\. D\. HunterMatplotlib: a 2D graphics environment\.Computing in Science & Engineering9\(3\),pp\. 90–95\.External Links:[Document](https://dx.doi.org/10.1109/MCSE.2007.55)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Hutter \(2006\)M\. HutterThe human knowledge compression prize\.Note:[http://prize\.hutter1\.net](http://prize.hutter1.net/)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1)\.
- Jaimovitch\-Lópezet al\.\(2021\)G\. Jaimovitch\-López, D\. Castellano Falcón, C\. Ferri, and J\. Hernández\-OralloThink big, teach small: do language models distil Occam’s razor?\.InAdvances in Neural Information Processing Systems 34 \(NeurIPS 2021\),pp\. 1610–1623\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1)\.
- Jianget al\.\(2024\)M\. Jiang, K\. Z\. Liu, M\. Zhong, R\. Schaeffer, S\. Ouyang, J\. Han, and S\. KoyejoInvestigating data contamination for pre\-training language models\.arXiv preprint arXiv:2401\.06059\.Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1)\.
- Jinet al\.\(2023\)E\. Jin, J\. Hu, Z\. Huang, R\. Zhang, J\. Wu, L\. Fei\-Fei, and R\. Martín\-MartínMini\-behavior: a procedurally generated benchmark for long\-horizon decision\-making in embodied ai\.arXiv preprint arXiv:2310\.01824\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Kipniset al\.\(2024\)A\. Kipnis, K\. Voudouris, L\. M\. S\. Buschoff, and E\. Schulzmetabench–A sparse benchmark to measure general ability in large language models\.arXiv preprint arXiv:2407\.12844\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Kluyveret al\.\(2016\)T\. Kluyver, B\. Ragan\-Kelley, F\. Pérez, B\. Granger, M\. Bussonnier, J\. Frederic, K\. Kelley, J\. Hamrick, J\. Grout, S\. Corlay, P\. Ivanov, D\. Avila, S\. Abdalla, C\. Willing, and Jupyter Development TeamJupyter Notebooks – a publishing format for reproducible computational workflows\.InPositioning and Power in Academic Publishing: Players, Agents and Agendas,F\. Loizides and B\. Schmidt \(Eds\.\),pp\. 87–90\.External Links:[Document](https://dx.doi.org/10.3233/978-1-61499-649-1-87)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Laloret al\.\(2024\)J\. P\. Lalor, P\. Rodriguez, J\. Sedoc, and J\. Hernandez\-OralloItem response theory for natural language processing\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts,pp\. 9–13\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Laloret al\.\(2016\)J\. P\. Lalor, H\. Wu, and H\. YuBuilding an evaluation scale using item response theory\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,Austin, Texas,pp\. 648–657\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1062)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Legg and Hutter \(2007\)S\. Legg and M\. HutterUniversal intelligence: a definition of machine intelligence\.Minds and Machines17\(4\),pp\. 391–444\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p1.1),[§8](https://arxiv.org/html/2610.14005#S8.p2.1)\.
- Legg and Veness \(2013\)S\. Legg and J\. VenessAn approximation of the universal intelligence measure\.InAlgorithmic Probability and Friends: Bayesian Prediction and Artificial Intelligence,LNCS, Vol\.7070,pp\. 236–249\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p6.1)\.
- Leike and Hutter \(2015\)J\. Leike and M\. HutterBad universal priors and notions of optimality\.InProceedings of the 28th Conference on Learning Theory \(COLT\),pp\. 1244–1259\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px2.p1.1)\.
- Levin \(1973\)L\. A\. LevinUniversal sequential search problems\.Problems of Information Transmission9\(3\),pp\. 265–266\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1),[Appendix F](https://arxiv.org/html/2610.14005#A6.p5.1),[§1](https://arxiv.org/html/2610.14005#S1.p1.1),[§1](https://arxiv.org/html/2610.14005#S1.p3.1)\.
- Levin \(2003\)L\. A\. LevinThe tale of one\-way functions\.Problems of Information Transmission39\(1\),pp\. 92–103\.Cited by:[§L\.2](https://arxiv.org/html/2610.14005#A12.SS2.p8.1),[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1)\.
- Levin \(2013\)L\. A\. LevinUniversal heuristics: how do humans solve “unsolvable” problems?\.InAlgorithmic Probability and Friends\. Bayesian Prediction and Artificial Intelligence: Papers from the Ray Solomonoff 85th Memorial Conference, Melbourne, VIC, Australia, November 30–December 2, 2011,pp\. 53–54\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.14005#S1.p3.1)\.
- Li and Vitányi \(2019\)M\. Li and P\. VitányiAn introduction to Kolmogorov complexity and its applications\.4th edition,Springer\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p2.1)\.
- Li \(2026\)M\. LiFrom Levin’s universal search to policy\-guided tree search\.Entropy28\(4\),pp\. 434\.Cited by:[§8](https://arxiv.org/html/2610.14005#S8.p3.1)\.
- Lord \(1975\)F\. M\. LordThe ‘ability’ scale in item characteristic curve theory\.Psychometrika40\(2\),pp\. 205–217\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Luccioniet al\.\(2025\)S\. Luccioni, Y\. Jernite, R\. Pierrard, I\. Moutawwakil, M\. Mitchell, B\. Gamazaychikov, J\. Qu, B\. Bi, M\. Weimann, H\. Downey, S\. Chamberlin, S\. Hooker, C\. Wu, and E\. StrubellAI Energy Score: initiative to establish comparable energy efficiency ratings for AI models\.Note:[https://huggingface\.github\.io/AIEnergyScore](https://huggingface.github.io/AIEnergyScore)Documentation, accessed 23 September 2026Cited by:[Appendix O](https://arxiv.org/html/2610.14005#A15.p3.1)\.
- Maaschet al\.\(2025\)J\. Maasch, J\. Kalantari, and K\. KhezeliCausalarc: abstract reasoning with causal world models\.arXiv preprint arXiv:2509\.03636\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Mahoney \(1999\)M\. V\. MahoneyText compression as a test for artificial intelligence\.AAAI/IAAI970\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1)\.
- Maia Poloet al\.\(2024\)F\. Maia Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. YurochkinTinyBenchmarks: evaluating LLMs with fewer examples\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 34303–34326\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Martínez\-Plumedet al\.\(2019\)F\. Martínez\-Plumed, R\. B\. Prudêncio, A\. Martínez\-Usó, and J\. Hernández\-OralloItem response theory in ai: analysing machine learning classifiers at the instance level\.Artificial intelligence271,pp\. 18–42\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p7.1)\.
- McCarthy \(2006\)J\. McCarthyChallenges to machine learning: relations between reality and appearance\.InInternational Conference on Inductive Logic Programming,pp\. 2–9\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.14005#S1.p1.1)\.
- Michell \(1999\)J\. MichellMeasurement in psychology: a critical history of a methodological concept\.Vol\.53,Cambridge University Press\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Mirzadehet al\.\(2025\)S\. I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. FarajtabarGSM\-symbolic: understanding the limitations of mathematical reasoning in large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.Human\-level control through deep reinforcement learning\.nature518\(7540\),pp\. 529–533\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Morriset al\.\(2023\)J\. X\. Morris, W\. Zhao, J\. T\. Chiu, V\. Shmatikov, and A\. M\. RushLanguage model inversion\.arXiv:2311\.13647\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p8.1)\.
- OpenAI \(2025\)OpenAIGPT\-5 system card\.Note:[https://openai\.com/index/gpt\-5\-system\-card/](https://openai.com/index/gpt-5-system-card/)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- OpenAI \(2026\)OpenAIGPT\-5\.6 system card\.Note:[https://deploymentsafety\.openai\.com/gpt\-5\-6](https://deploymentsafety.openai.com/gpt-5-6)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Orseauet al\.\(2024\)L\. Orseau, M\. Hutter, and L\. H\. LelisExponential speedups by rerooting Levin tree search\.arXiv preprint arXiv:2412\.05196\.Cited by:[§8](https://arxiv.org/html/2610.14005#S8.p3.1)\.
- Paglieriet al\.\(2025\)D\. Paglieri, B\. Cupiał, S\. Coward, U\. Piterbarg, M\. Wolczyk, A\. Khan, E\. Pignatelli, Ł\. Kuciński, L\. Pinto, R\. Fergus, J\. N\. Foerster, J\. Parker\-Holder, and T\. RocktäschelBALROG: benchmarking agentic LLM and VLM reasoning on games\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fp6t3F669F)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 8024–8035\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Pérezet al\.\(2021\)J\. Pérez, P\. Barceló, and J\. MarinkovicAttention is Turing\-complete\.Journal of Machine Learning Research22\(75\),pp\. 1–35\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p4.1)\.
- Pérez\-Liébanaet al\.\(2019\)D\. Pérez\-Liébana, S\. M\. Lucas, R\. D\. Gaina, J\. Togelius, A\. Khalifa, and J\. LiuGeneral video game artificial intelligence\.Synthesis Lectures on Games and Computational Intelligence,Morgan & Claypool,San Rafael, CA\.External Links:[Document](https://dx.doi.org/10.2200/S00944ED1V01Y201908GCI005)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Phanet al\.\(2025\)L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, S\. Shi, M\. Choi, A\. Agrawal, A\. Chopra,et al\.Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Qwen Team \(2024\)Qwen TeamQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Rustet al\.\(2021\)J\. Rust, M\. Kosinski, and D\. StillwellModern psychometrics: the science of psychological assessment\.4th Edition, Routledge\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Schlangen \(2021\)D\. SchlangenTargeting the benchmark: on methodology in current natural language processing research\.InAssociation for Computational Linguistics,pp\. 670–674\.External Links:[Link](https://aclanthology.org/2021.acl-short.85/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-short.85)Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Schuurmans \(2023\)D\. SchuurmansMemory augmented large language models are computationally universal\.arXiv:2301\.04589\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p4.1)\.
- Shannon \(1948\)C\. E\. ShannonA mathematical theory of communication\.The Bell system technical journal27,pp\. 379–423\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Shannon \(1949\)C\. E\. ShannonCommunication theory of secrecy systems\.The Bell system technical journal28\(4\),pp\. 656–715\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1)\.
- Shannon \(1951\)C\. E\. ShannonPrediction and entropy of printed english\.Bell system technical journal30\(1\),pp\. 50–64\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px3.p1.1)\.
- Shinet al\.\(2020\)T\. Shin, Y\. Razeghi, R\. L\. Logan IV, E\. Wallace, and S\. SinghAutoPrompt: eliciting knowledge from language models with automatically generated prompts\.InProceedings of EMNLP 2020,Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p8.1)\.
- Solomonoff \(1960\)R\. J\. SolomonoffA preliminary report on a general theory of inductive inference\.Zator Company Cambridge, MA\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Solomonoff \(1964a\)R\. J\. SolomonoffA formal theory of inductive inference\. Part I\.Information and control7\(1\),pp\. 1–22\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Solomonoff \(1964b\)R\. J\. SolomonoffA formal theory of inductive inference\. Part II\.Information and control7\(2\),pp\. 224–254\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1)\.
- Sterkenburg \(2016\)T\. F\. SterkenburgSolomonoff prediction and occam’s razor\.Philosophy of Science83\(4\),pp\. 459–479\.Cited by:[§4](https://arxiv.org/html/2610.14005#S4.p4.1)\.
- Stevens \(1946\)S\. S\. StevensOn the theory of scales of measurement\.Science103\(2684\),pp\. 677–680\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Team OLMoet al\.\(2024\)Team OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, A\. Ettinger, M\. Guerquin, D\. Heineman, H\. Ivison, P\. W\. Koh, J\. Liu, S\. Malik, W\. Merrill, L\. J\. V\. Miranda, J\. Morrison, T\. Murray, C\. Nam, J\. Poznanski, V\. Pyatkin, A\. Rangapur, M\. Schmitz, S\. Skjonsberg, D\. Wadden, C\. Wilhelm, M\. Wilson, L\. Zettlemoyer, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi2 OLMo 2 furious\.arXiv preprint arXiv:2501\.00656\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p2.1)\.
- Thurstone \(1937\)L\. L\. ThurstoneAbility, motivation, and speed\.Psychometrika2\(4\),pp\. 249–254\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p1.1),[§4](https://arxiv.org/html/2610.14005#S4.p1.1)\.
- Trabin and Weiss \(1983\)T\. E\. Trabin and D\. J\. WeissThe person response curve: fit of individuals to item response theory models\.InNew horizons in testing,pp\. 83–108\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Tueroet al\.\(2026\)J\. Tuero, M\. Buro, L\. Orseau, and L\. H\. S\. LelisStructure\-induced information for rerooting Levin tree search\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§8](https://arxiv.org/html/2610.14005#S8.p3.1)\.
- Vale and Weiss \(1975\)C\. D\. Vale and D\. J\. WeissA study of computer\-administered stradaptive ability testing\.\.Technical reportMinnesota Univ\. Minneapolis Dept\. of Psychology\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Vaniaet al\.\(2021\)C\. Vania, P\. M\. Htut, W\. Huang, D\. Mungra, R\. Y\. Pang, J\. Phang, H\. Liu, K\. Cho, and S\. R\. BowmanComparing test sets with item response theory\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),pp\. 1141–1158\.External Links:[Link](https://aclanthology.org/2021.acl-long.92),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.92)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px4.p1.1)\.
- Virtanenet al\.\(2020\)P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright,et al\.SciPy 1\.0: fundamental algorithms for scientific computing in Python\.Nature methods17\(3\),pp\. 261–272\.Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1),[footnote 3](https://arxiv.org/html/2610.14005#footnote3)\.
- Voudouriset al\.\(2025\)K\. Voudouris, B\. Slater, L\. G\. Cheke, W\. Schellaert, J\. Hernández\-Orallo, M\. Halina, M\. Patel, I\. Alhas, M\. G\. Mecattaf, J\. Burden,et al\.The animal\-ai environment: a virtual laboratory for comparative cognition and artificial intelligence research\.Behavior Research Methods57\(4\),pp\. 107\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1)\.
- Wallmarket al\.\(2024\)J\. Wallmark, M\. Josefsson, and M\. WibergIntroducing flexible monotone multiple choice item response theory models and bit scales\.arXiv preprint arXiv:2410\.01480\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px6.p1.1)\.
- Wenet al\.\(2023\)Y\. Wen, N\. Jain, J\. Kirchenbauer, M\. Goldblum, J\. Geiping, and T\. GoldsteinHard prompts made easy: gradient\-based discrete optimization for prompt tuning and discovery\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p8.1)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 38–45\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[Appendix R](https://arxiv.org/html/2610.14005#A18.p3.1)\.
- Yinget al\.\(2025\)L\. Ying, K\. M\. Collins, P\. Sharma, C\. Colas, K\. I\. Zhao, A\. Weller, Z\. Tavares, P\. Isola, S\. J\. Gershman, J\. D\. Andreas, T\. L\. Griffiths, F\. Chollet, K\. R\. Allen, and J\. B\. TenenbaumAssessing adaptive world models in machines with novel games\.arXiv preprint arXiv:2507\.12821\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1),[§8](https://arxiv.org/html/2610.14005#S8.p2.1)\.
- Yinget al\.\(2026\)L\. Ying, R\. Truong, P\. Sharma, K\. I\. Zhao, N\. Cloos, K\. R\. Allen, T\. L\. Griffiths, K\. M\. Collins, J\. Hernández\-Orallo, P\. Isola, S\. J\. Gershman, and J\. B\. TenenbaumAI gamestore: scalable, open\-ended evaluation of machine general intelligence with human games\.arXiv preprint arXiv:2602\.17594\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1),[§8](https://arxiv.org/html/2610.14005#S8.p2.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§1](https://arxiv.org/html/2610.14005#S1.p2.1),[§2](https://arxiv.org/html/2610.14005#S2.p5.1)\.
- Zenilet al\.\(2018\)H\. Zenil, S\. Hernández\-Orozco, N\. A\. Kiani, F\. Soler\-Toscano, A\. Rueda\-Toicen, and J\. TegnérA decomposition method for global evaluation of Shannon entropy and local estimations of algorithmic complexity\.Entropy20\(8\),pp\. 605\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p6.1)\.
- Zhenget al\.\(2025\)H\. Zheng, W\. Chu, B\. Zhang, Z\. Wu, A\. Wang, B\. Feng, C\. Zou, Y\. Sun, N\. Kovachki, Z\. Ross,et al\.Inversebench: benchmarking plug\-and\-play diffusion priors for inverse problems in physical sciences\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 90912–90940\.Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2026\)Y\. Zhou, A\. Xu, Y\. Zhou, J\. Singh, J\. Gui, and S\. JotyVariation in verification: understanding verification dynamics in large language models\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 144597–144642\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p7.1)\.
- Zirviet al\.\(2025\)R\. Zirvi, B\. Tolooshams, and A\. AnandkumarDiffusion state\-guided projected gradient for inverse problems\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=kRBQwlkFSP)Cited by:[Appendix S](https://arxiv.org/html/2610.14005#A19.SS0.SSS0.Px1.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. FredriksonUniversal and transferable adversarial attacks on aligned language models\.arXiv:2307\.15043\.Cited by:[§2](https://arxiv.org/html/2610.14005#S2.p8.1)\.

## Appendix contents

## Appendix AFull Results for Qwen3\-14B

Here we include some other plots and tables for Qwen3\-14B specifically that did not fit in the main paper\. Let us start with the detailed results from the 12 models we used as examinees\. More precisely, they were 15, as we tried two versions of Sonnet, Opus and Fable, because on a first attempt they had a high degree of refusal, as shown in[Table5](https://arxiv.org/html/2610.14005#A1.T5)\. Our guess for this high level of refusal is that some items look cryptic to the safety filters, as if they were jailbreaks\. We had three options: remove most of the Anthropic family, remove the items affected or include them with a clear indication that they are routing items to other models\. We explored the last two111Refusals are not spread evenly\. The plain Anthropic models refused at least once on 22 of the 51 items certified under Qwen3\-14B, with a medianK​t^\|\\widehat\{Kt\}^\{\|\}of 176 bits against 45 for the rest, including 8 of the 10 items above 190 bits\. Treating the starred models’ non\-answers as missing, instead of as failures, leaves the correlation forK​t^\|\\widehat\{Kt\}^\{\|\}unchanged, at−0\.79\-0\.79\., but include the latter in the paper: we kept the version with fallback for all the experiments in the paper, shown with a star\.[Table5](https://arxiv.org/html/2610.14005#A1.T5)includes all the results, finalising all the runs for the non\-fallback versions, for completeness222The plain Claude Sonnet 5 answered its first 54 items with a budget of 8000 tokens, and the rest with 16000\. Its fallback version ran all items with 16000, so their counts of cut replies are not directly comparable, but the increase made no difference to how often it was cut\.\. It also shows that some of the outputs that were still cut by budget or refusal did not disappear \(especially for Sonnet\*\)\. The discrepancy of budget comes from the first runs trying to determine what an appropriate budget could be\. Where we could test it, increasing the budget did not have an effect\. In particular, for Sonnet 5, doubling the budget from 8000 to 16000 did not reduce the share of cut replies, 23% against 25%, so a larger budget is unlikely to change its results\. GPT\-5 nano was run only at 8000, but we followed our criterion of assigning a budget roughly proportional to the potential and needs of the model\. In the era of reasoning models, an examinee is a pair model and budget\. In this case, increasing the budget for any of the models would have not had a noticeable effect\. The models and budgets are sufficient for the purpose of our research questions \(such as the comparison with ECI\)\.

Table 5:Reason of non\-answers for the 71 valid, unquestionable items under Qwen3\-14B\. We tried two versions of sonnet, opus and fable to reduce refusal, and we kept the one with fallback, marked with a star, in which a refused request is answered by another model that Anthropic recommends\. Comparing the two versions suggests that the fallback served roughly two fifths of the replies of Fable 5\.1\*, a tenth of those of Opus 5\*, and almost none of Sonnet 5\*’s\. These models can change their delegation strategies, so these results may change in future runs\. All scores, plots and calculations are performed with these 12 models, and we consider any of these reasons \(cut, refusal, timeout, malformed or other\) as a failure\. A reply is cut when it reaches the token budget before the model finishes, and ‘cut’ only counts those that were cutandgave no answer\. A refusal or a truncation is not evidence that the model could not solve the item, given more tokens or finding the cause for the rejection\. The last column is the score on the rows that were answered\.In[Table6](https://arxiv.org/html/2610.14005#A1.T6)we show the scores of the 12 models for all unquestionable items following different criteria\. In general, in this paper, we usefull:mean, i\.e\., the average of the three attempts, each of them considered as the continuation of the sequence must agree fully\.

Table 6:Six different ways of scoring for the 71 valid, unquestionable items under Qwen3\-14B\.prefix%gives the percentage of terms before its first mistake\.fullscores the same attempt all or nothing: every term right, or none of the credit\. Within each, as we have three repetitions when evaluating,worstandbesttake the least and the most of the three attempts andmeanaverages them\.agreesays whether the three said the same thing, right or wrong\. An attempt with no reply is left out of the agreement, as three refusals are not really agreeing on anything\.[Table7](https://arxiv.org/html/2610.14005#A1.T7)shows the distribution grid of its 71 unquestionable items by measured \(r^,t^\\hat\{r\},\\hat\{t\}\)\. We see that most items are concentrated in low and medium transformations and low retrieval\.

Table 7:Number of*valid*/*unquestionable*/*certified*items under Qwen3\-14B, by the cell they fell in\.[Table8](https://arxiv.org/html/2610.14005#A1.T8)shows Spearman correlations for a wide range of metrics per model and together for the 51 certified items for Qwen3\-14B\. It is an extension of[Table3](https://arxiv.org/html/2610.14005#S7.T3), but here we see that the correlations ofr0r^\{0\}andt0t^\{0\}are very similar to the measured ones\. We also see that that length of the programs in number of characters \(ℓc​h​a​r\\ell\_\{char\}\) is a good proxy, only clearly improved by the last three rows\.

The length of the seed and other programs in the process is informative only on items the pipeline has validated and made unquestionable, and the correlation grows along the process that filters usingVVverification andℳ\\mathcal\{M\}coding\. Certification also shows that a weak machine can run a description of each item, which seed length cannot\.K​t^\|\\widehat\{Kt\}^\{\|\}is also a measure and not a proxy\. It is in bits under a stated reference machine, conditional on the prefix, and a minimum over many descriptions, so it does not depend on how verbose one description is, and it can be compared across batteries and composers\. Among the harder half of the items, above 101 bits, it also correlates more strongly with the accuracy of the six strongest examinees, at−0\.48\-0\.48against−0\.31\-0\.31, which is the supra region we are most interested in\.

Table 8:Full version of[Table3](https://arxiv.org/html/2610.14005#S7.T3)\. Spearman correlations between several metrics and full correct response for the 51 valid, unquestionable and certified items under Qwen3\-14B\. Each item had three repetitions per examinee\.[Figure5](https://arxiv.org/html/2610.14005#A1.F5)\(left\) shows the empirical curves of the points for the 51 certified examples for Qwen3\-14B\. It shows a clean monotone decline of performance versus Levin’sK​t\|Kt^\{\|\}\. The two local models fall after the second bin, On the right we see the histogram of difficulties \(K​t\|Kt^\{\|\}\)\. The first 100 bits can be explained by the fact that very simple items that have a full rule around 50\-100 bits to express the sequence in the unconditional regime collapse when the shortest conditional program is searched for\. Many items end up very close to zero, as they are continued by the model without any other information\. For these items,log2⁡τ\\log\_\{2\}\\tauadds very few bits in any case\.

Figure 5:Left: Empirical agent characteristic curves \(same\-size bins\) for the 12 LLM, withK​t\|Kt^\{\|\}provided by Qwen3\-14B \(51 certified examples\)\. Right: a histogram of difficulty \(K​t\|Kt^\{\|\}\) of the battery\.
## Appendix BResults for Qwen3\-8B

Here we include all the plots and tables for Qwen3\-8B specifically\. This model is weaker and it has a smaller number of certified items\.[Table9](https://arxiv.org/html/2610.14005#A2.T9)shows the distribution grid of its 80 unquestionable items by measured \(r^,t^\\hat\{r\},\\hat\{t\}\)\. We see that, again, most items are concentrated in low and medium transformations and low retrieval\. For 29 of the 107 valid items, Qwen3\-8B was costed with an earlier value of the clause constant,δ=13\.3\\delta=13\.3bits\. Recomputed from the stored margins withδ=22\.7\\delta=22\.7, two of them become questionable, neither of them certified, and one certified item,0002, needs a prefix of 13 terms instead of 5\. The results in this appendix use the values as costed\.

Table 9:Number of*valid*/*unquestionable*/*certified*items under Qwen3\-8B, by the cell they fell in\.For Qwen3\-8B specifically,[Figure6](https://arxiv.org/html/2610.14005#A2.F6)shows the percentage of items being solved by each examinee for each cell in the grid\. Again, highttwith lowrrseems to be a specific area where models tend to fail\.

![Refer to caption](https://arxiv.org/html/2610.14005v1/fig_heat_unq_fine_full_qwen3-8b_bare.png)Figure 6:The 12 LLM examinees on the \(r^,t^\\hat\{r\},\\hat\{t\}\) grid and accuracy for the unquestionable examples for Qwen3\-8B\.[Figure7](https://arxiv.org/html/2610.14005#A2.F7)shows the logistic fit \(left\) of the points for the 32 certified examples for Qwen3\-8B \(2​N2Nprior points added at \(0,1\) as anchor\)\. The right plot compares the ability of each model with Item Response Theory estimates of the abilities of the same models according to Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\.

Figure 7:Left: Agent characteristic curves for the 12 LLM, withK​t\|Kt^\{\|\}provided by Qwen3\-8B \(32 certified examples\), with a logistic fit, where the ability of each modelθ\\theta, measured in bits, is estimated by the point where performance is 0\.5\. Right: Comparison between each model’sabilityand the psychometric, populational score given by Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\. Only 10 of our 12 are listed in ECI\.[Table10](https://arxiv.org/html/2610.14005#A2.T10)shows the correlations between the metrics and the scores of the 12 LLMs \(plus all the scores together\) for Qwen3\-8B\. The first thing we observe is that the magnitude of the correlations is smaller than those with Qwen3\-14B, especially for the most powerful models \(it is−0\.27\-0\.27for gpt5sol\)\. This is clearly explained by the lowerK​t\|Kt^\{\|\}estimated by Qwen3\-8B, which is very visible in the histogram of the right of[Figure9](https://arxiv.org/html/2610.14005#A2.F9)\. One thing to highlight is thatK​c\|Kc^\{\|\}achieves better results thanK​t\|Kt^\{\|\}at the lower end, possibly because may items are certified with an empty program, andτ\\taucannot distinguish between them, while thec\(x1:n\)c\(x\_\{1:n\}\)component ofK​t\|Kt^\{\|\}can give more refinement at this lower end\. But in this reference machine, the simple character lengths of the seed and other programs collapse, especially for the frontier models\.

Table 10:Spearman correlations between several metrics and full correct response for the 32 valid, unquestionable and certified items under Qwen3\-8B\. Each item had three repetitions per examinee\.[Figure8](https://arxiv.org/html/2610.14005#A2.F8)shows how theK​t\|Kt^\{\|\}metric changes between two reference machines of different family\. Not only are many items not certified by Qwen3\-8B, as compared to Phi\-4\-14B\-reasoning, but their difficulty is higher\. Qwen3\-8B is a worse compressor, but mostly because many equivalent programs and hints that reduceK​t\|Kt^\{\|\}in Phi\-4\-14B\-reasoning do not yield the sequence for Qwen3\-8B\. This adds to our answer to RQ6 in how the power of the reference machine affects the results\.

Figure 8:Comparison ofK​t\|Kt^\{\|\}between two reference machines \(Qwen\-3\-8B and Phi\-4\-14B\-reasoning\) versus the average results of 12 other LLMs on the unquestionable items in the battery\. As predicted, success decreases with the theoretical complexity of the item\. The weaker model \(Qwen\-3\-8B, purple\) brings more refinement in the lower end, but it is not able to cost many items on the upper end\. An arrow pointing left means Phi\-4\-14B\-reasoning describes the item at less cost\. Usually Phi\-4\-14B\-reasoning is a more effective coder and achieves lower values\. Values nearK​t=0Kt=0represent cases where the reference machine was able to continue the sequence correctly without any program or hint\.[Figure9](https://arxiv.org/html/2610.14005#A2.F9)\(left\) shows the empirical curves of the points for the 32 certified examples for Qwen3\-8B\. It shows a less clean monotone decline of performance versus Levin’sK​t\|Kt^\{\|\}\. The two local models fall after the second bin, but the rest have a comeback for the last two quantiles\. Very powerful models do not go below 0\.8\. In reality, we are actually seeing the beginning of a much longer characteristic curve that this model cannot really estimate for frontier models\. On the right we see the histogram of difficulties \(K​t\|Kt^\{\|\}\), and that explains the figure\. Qwen3\-8B really does not have sensitivity for large difficulty and powerful models\.

Figure 9:Left: Empirical agent characteristic curves \(same\-size bins\) for the 12 LLM, withK​t\|Kt^\{\|\}provided by Qwen3\-8B \(32 certified examples\)\. Right: a histogram of difficulty \(K​t\|Kt^\{\|\}\) of the battery\.
## Appendix CResults for Phi\-4\-14B\-reasoning

Here we include all the plots and tables for Phi\-4\-14B\-reasoning specifically\. This model situates itself in between the previous two models\.[Table11](https://arxiv.org/html/2610.14005#A3.T11)shows the distribution grid of its 80 unquestionable items by measured \(r^,t^\\hat\{r\},\\hat\{t\}\)\. We see, once again, that most items are concentrated in low and medium transformations and low retrieval\.

Table 11:Number of*valid*/*unquestionable*/*certified*items under Phi\-4\-14B\-reasoning, by the cell they fell in\.Following with Phi\-4\-14B\-reasoning,[Figure10](https://arxiv.org/html/2610.14005#A3.F10)shows the percentage of items being solved by each examinee for each cell in the grid\. Similarly to the same figure for the other models, highttwith lowrrseems to be a specific area where models tend to fail\.

![Refer to caption](https://arxiv.org/html/2610.14005v1/fig_heat_unq_fine_full_phi-4-reasoning_bare.png)Figure 10:The 12 LLM examinees on \(r^,t^\\hat\{r\},\\hat\{t\}\) grid and accuracy for the unquestionable examples for Phi\-4\-reasoning\.[Figure11](https://arxiv.org/html/2610.14005#A3.F11)shows the logistic fit \(left\)\) of the points for the 38 certified examples for Phi\-4\-14B\-reasoning \(2​N2Nprior points added at \(0,1\) as anchor\)\. The right plot compares the ability of each model with Item Response Theory estimates of the abilities of the same models according to Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\. The correlations are very high again\.

Figure 11:Left: Agent characteristic curves for the 12 LLM, withK​t\|Kt^\{\|\}provided by Phi\-4\-14B\-reasoning \(38 certified examples\), with a logistic fit, where the ability of each modelθ\\theta, measured in bits, is estimated by the point where performance is 0\.5\. Right: Comparison between each model’sabilityand the psychometric, populational score given by Epoch’s Capability Index \(ECI\)\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\)\. Only 10 of our 12 are listed in ECI\.[Table12](https://arxiv.org/html/2610.14005#A3.T12)shows the correlations with the metrics andK​t\|Kt^\{\|\}as derived by Phi\-4\-reasoning\. The correlations are a bit in between Qwen3\-8B and Qwen3\-14B, which is consistent with a range of difficulties that is also in between\. For this case, the dominance ofK​tKtoverK​cKcis very noticeable, and no metric really competes withK​tKt\. In this reference machine, the simple character lengths of the seed and other programs are left about 0\.20 points of correlation overall belowK​t\|Kt^\{\|\}, again especially well behind for the frontier models\.

Table 12:Spearman correlations between several metrics and full correct response for the 38 valid, unquestionable and certified items under Phi\-4\-reasoning\. Each item had three repetitions per examinee\.[Figure12](https://arxiv.org/html/2610.14005#A3.F12)shows how theK​t\|Kt^\{\|\}metric changes between two reference machines of a different family\. Not only are a few items not certified by Phi\-4\-14B\-reasoning, as compared to Qwen3\-14B, but their difficulty is higher\. Phi\-4\-14B\-reasoning is a worse compressor, but in this cases it is not for those items that both certify \(arrows move similarly in both direction\) but almost ony because many equivalent programs and hints that reduceK​t\|Kt^\{\|\}in Qwen\-3\-14b do not yield the sequence for Phi\-4\-14B\-reasoning\. This completes our answer to RQ6 in how the power of the reference machine affects the results\.

Figure 12:Comparison ofK​t\|Kt^\{\|\}between two reference machines \(Qwen\-3\-14B and Phi\-4\-14B\-reasoning\) versus the average results of 12 other LLMs on the unquestionable items in the battery\. As predicted, success decreases with the theoretical complexity of the item\. The weaker model \(Phi\-4\-14B\-reasoning, purple\) brings more refinement in the lower end, but it is not able to cost many items on the upper end\. An arrow pointing left means Qwen\-3\-14B describes the item at less cost\. Here, we do not see the dominance of Qwen\-3\-14B and some arrows go left \(in green\), but some others go right \(in red\)\. Values nearK​t=0Kt=0represent cases where the reference machine was able to continue the sequence correctly without any program or hint\.Finally,[Figure13](https://arxiv.org/html/2610.14005#A3.F13)\(left\) shows the empirical curves of the points for the 38 certified examples for Phi\-4\-14B\-reasoning\. It shows an irregular, yet flat behaviour of performance until the last bin of Levin’sK​t\|Kt^\{\|\}\. The two local models fall after the second bin, but the rest only fall in the last bin, showing that the critical part for many models happens around difficulty 112\. On the right we see the histogram of difficulties \(K​t\|Kt^\{\|\}\), and the area around 112 is where we have most spread\. It is not only that this reference machine can only reach 187 as maximumK​t\|Kt^\{\|\}but also that the spread and number of difficulties make the curves less smooth they should be\.

Figure 13:Left: Empirical agent characteristic curves \(same\-size bins\) for the 12 LLM, withK​t\|Kt^\{\|\}provided by Phi\-4\-14B\-reasoning \(38 certified examples\)\. Left: empirical curve with same\-size bins\. Right: a histogram of difficulty \(K​t\|Kt^\{\|\}\) of the battery\.
## Appendix DRatio Scales

IfK​t^\|\\widehat\{Kt\}^\{\|\}gives a ratio scale, abilities measured with different reference machines should differ only by a unit, that is, be proportional, as temperatures in kelvin and rankine are, and not merely related by a line with an offset, as Celsius and Fahrenheit are\. The Pearson correlations between the abilities of the twelve examinees under the three machines are 0\.985 or higher \(Table[13](https://arxiv.org/html/2610.14005#A4.T13)\)\. Fitting a line between each pair of machines, the intercepts are compatible with zero, at\+7\+7,\+1\+1and−3\-3bits, with 95% bootstrap intervals over examinees that include zero, and forcing the line through the origin leaves the fit essentially unchanged, withR2R^\{2\}of 0\.988, 0\.971 and 0\.971 against 0\.990, 0\.971 and 0\.972\. Abilities under Qwen3\-8B are about 1\.46 times those under Qwen3\-14B, since the weaker machine describes the same items with more bits, and under Phi\-4\-reasoning they are about the same as under Qwen3\-14B\. Without the two small examinees, whose abilities are near zero under every machine, the correlations are 0\.94 to 0\.98 and the intercepts remain compatible with zero, although their intervals are wider\.

Table 13:Pearson correlations between the abilities of the twelve examinees under the three reference machines\. This supports the idea of the metric being a ratio scale, with only the coefficient changing between machines\. See[AppendixD](https://arxiv.org/html/2610.14005#A4)for justification about the common zero\.This is just an empirical check on the possibility of the metric being a ratio scale, which was not set as a research question of the paper\. As future work, experimental comparison in terms of training and inference compute \(scaling laws\) with this metric could be performed, to see if the behaviour between ability and the logarithm of a meaningful combination of these two elements of compute could be additive\.

## Appendix EThe Reference Machine as an Examinee

The choice of a reference machineℳ\\mathcal\{M\}forK​t^ℳ\\widehat\{Kt\}\_\{\\mathcal\{M\}\}tries to capture human language and culture, but still the test ends up biased by the particularities ofℳ\\mathcal\{M\}\. But how much biased the test is compared to other reference machines? Of course, in our experiments we did not use the three reference machines as examinees, because this would have contaminated the averages across models, and the overall correlations\. But it is now time to see how much a reference machine favours itself, and this will indirectly bring some light the question about the particularities of different reference machines\. On the key results, there are things we can observe without any extra experiment\. Namely, the choice of reference machine barely changes how the examinees are ranked\. Their abilities under the three machines agree at Spearman\+0\.99\+0\.99,\+0\.92\+0\.92and\+0\.95\+0\.95\(Appendix[F](https://arxiv.org/html/2610.14005#A6)\)\.

But the effect must appear when applied to itself\. A trivial question would b whether a machine does well on the items it certifies\. Yet the answer to this questions is known in advance\. An item enters a machine’s battery only if that machine ran some description of it successfully, so the battery is selected to be solvable by that machine\. This is a contamination by construction, and it is why the reference machines are never scored alongside the other examinees in the body of the paper\.

Instead, to get more insight on this phenomenon we thought of paired analysis\. We can use the items that two machines both certify and give them to both machines\. Every item then has a value ofK​t^\\widehat\{Kt\}under each machine and a result as an examinee\. The question it answers is whether a machine’s ownK​t^\\widehat\{Kt\}orders its answers better than the other machine’sK​t^\\widehat\{Kt\}does\. Because the question is symmetric, it needs both machines\.

We chose Qwen3\-14B and Phi\-4\-reasoning because they are the two models with similar number of parameters and the capability is close\. Then we select the 28 items that both Qwen3\-14B and Phi\-4\-reasoning certify\. These usually have lowK​t\|Kt^\{\|\}\. The medianK​t\|^\\widehat\{Kt^\{\|\}\}over them is 13 bits under Qwen3\-14B and 24 bits under Phi\-4\-reasoning\. Six of the twelve other examinees answer more than 90% of them correctly\. It is important to understand that this is a special set of items\.

Also, to understand if the comparison is fair, we have to explain how the two machines were run\. As said above, both machines answered the same 28 items with the same prompt as every other examinee \(Figure[14](https://arxiv.org/html/2610.14005#A9.F14)\)\. Each was shown the same prefix, the longest that any of the three reference machines needed for that item, and asked for the nextkf=5k\_\{f\}=5terms, as all other examinees\. Both were loaded in 4\-bit NF4 with reasoning enabled and used greedy decoding, as were used as reference machines\. These are the coincidences\. Now the differences\. The two runs differed how the end token is identified and the token budget, in the respects listed in Table[14](https://arxiv.org/html/2610.14005#A5.T14)\. The stopping rule was needed for Phi\-4\-reasoning\. Without it, every reply ran to the full budget of 8000 tokens, because the model reaches the answer early and then keeps restating it\. To reduce the running time, we implemented a rule that stops a reply as soon as the five terms appear as a run of comma\-separated fields on one line, or once a passage of 200 characters has been repeated five times after the first 2000 tokens\. The smaller budget on 15 items was needed to fit the second part of the run into the time available\.

Table 14:The conditions under which the two reference machines were run as examinees\. The rows that differ are explained in the text\.As we ran the two machines with greedy decoding, the result is deterministic, so we stopped running three \(unnecessary\) repetitions at some point\. Repetitions would only measure the judge, since the replies themselves do not change\. But for our exploration, the repetitions were almost always equivalent, so we decided not to use extra compute on this\.

The differences in Table[14](https://arxiv.org/html/2610.14005#A5.T14)pull in opposite directions\. The stopping rule is more lenient to Phi\-4\-reasoning, since it ends a reply at the point where the answer appears, while the smaller budget is less lenient\. We cannot say which effect is larger\.

Under this rule Qwen3\-14B solves 21 of the 28 items and Phi\-4\-reasoning 15\. As a reference machine, Qwen3\-14B continued 15 of the 28 with the empty description, and as an examinee it solves 14 of those; the exception is0005\. Phi\-4\-reasoning continued 14 of the 28 with the empty description and solves 10 of those\. The prefix is not the reason for the four it fails, since each was shown at least as many terms as it had needed\. A likely cause is the prompt\. As a reference machine the model continues a sequence placed after the word*Rule*\. As an examinee, the prompt mentions it maythinkbefore answering, and this model then reasons at length, exceeding the budget more often than with the prompt that was used for verification in the annotator when used as a reference machine\.

Given all these considerations, we can now look at Table[15](https://arxiv.org/html/2610.14005#A5.T15), which lists the 28 items\. For each machine, we show the kind of description that certified the item \(with no help, i\.e\., ‘empty’, a partial indication, ‘hint’, a full ‘rule’, or the ‘seed’ the composer first wrote\), itsK​t\|^\\widehat\{Kt^\{\|\}\}, and whether the machine solves it as an examinee\.

Table 15:The 28 items certified by both machines, in order ofK​t\|^\\widehat\{Kt^\{\|\}\}under Qwen3\-14B\. For each machine the table gives the kind of description that certified the item, itsK​t\|^\\widehat\{Kt^\{\|\}\}in bits, and whether the machine solves it as an examinee\.Now we are ready to answer the question of how much the reference machine biases its own evaluation\. For each examinee we compute the Spearman correlation, over the 28 items, between its outcome on each item andK​t\|^\\widehat\{Kt^\{\|\}\}, once with Qwen3\-14B’s values and once with Phi\-4\-reasoning’s\. For the two reference machines the outcome is whether the item was solved \(1\) or not \(0\)\. For the twelve other examinees, which we also include, whose replies do vary between attempts, it is the proportion of their three attempts answered correctly\. A more negative correlation means that measure orders the items better for that examinee, in the sense that the items it calls more expensive are the ones the examinee fails\. Ties are given their average rank\.

Table 16:Spearman correlation between outcome andK​t\|^\\widehat\{Kt^\{\|\}\}over the 28 items, with each machine’s values ofK​t\|^\\widehat\{Kt^\{\|\}\}\. A more negative value means the measure orders the items better for that examinee\.Table[16](https://arxiv.org/html/2610.14005#A5.T16)gives the result\. For the twelve examinees that are not reference machines, shown in the bottom line, the two measures are almost equally good, at−0\.35\-0\.35and−0\.33\-0\.33\. But, for each reference machine, its own measure is the better one\. The margin is 0\.18 for Qwen3\-14B and 0\.07 for Phi\-4\-reasoning\.

We checked how much this depends on particular items\. Restricted to the 25 items that Qwen3\-8B also certifies, the pattern holds and Phi\-4\-reasoning’s margin widens\. Qwen3\-14B correlates−0\.46\-0\.46with its own measure and−0\.37\-0\.37with the other, and Phi\-4\-reasoning−0\.48\-0\.48with its own and−0\.34\-0\.34with the other\. Leaving out each of the 28 items in turn, Qwen3\-14B’s own measure is the better one every time, by at least 0\.10\. Phi\-4\-reasoning’s own measure is also the better one every time, but by very little when0126is left out\.

So we can conclude what we expected: a reference machine is favoured by its own battery, but we can quantify it and understand the reasons better\. The first reason is the selection described above\. A machine’s battery holds only items that the machine could run the given solution \(in many cases with the empty program\)\. The second reason is what we have shown\. On items that both machines certify, a machine’s ownK​t^\\widehat\{Kt\}orders its answers better than the other machine’sK​t^\\widehat\{Kt\}does\. The effect appears for both machines, in the expected direction, while the twelve other examinees find the two measures almost equally good\. It is clear for Qwen3\-14B and small for Phi\-4\-reasoning\.

The evidence is not conclusive about the magnitude because the 28 items are among the least expensive in the battery, so the range ofK​t^\\widehat\{Kt\}over them is narrow\. Also, as said, the two machines were run under conditions that differ in the ways listed in Table[14](https://arxiv.org/html/2610.14005#A5.T14)\. And there is one reference machine on each side, so the result rests on two comparisons\.

But we stay reassured about our decision that the reference machine should not be scored as an examinee on a battery it helped to build, both because of the selection bias performed by certification and because its own measure fits it better than any other machine\.

## Appendix FStatistical Tests

We report intervals and tests for the main quantitative results\. Intervals are at 95%\. For correlations over items we use a percentile bootstrap that resamples items, 4000 times, because the uncertainty comes from the composition of the battery\. For the interval of the Pearson correlation with ECI we use the Fisherzz\-transformation\. It covers the ten examinees that ECI lists, since OLMo\-2\-7B and Qwen2\.5\-1\.5B are not in it\. For a proportion we use the Clopper–Pearson interval\. Allpp\-values come from one\-sided permutation tests in the predicted direction, with 10,000 permutations over items and 20,000 over examinees333We implemented the tests directly, but we then used SciPy’s functions\([Virtanen et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib25)\)and replicated the results, with very minor differences:spearmanr,bootstrapwith the percentile method,permutation\_test, andbinomtestwith the exact interval, which is Clopper–Pearson\.\.

Table 17:Intervals and tests for the main results\. The first block corresponds to the last column of Tables[8](https://arxiv.org/html/2610.14005#A1.T8),[10](https://arxiv.org/html/2610.14005#A2.T10)and[12](https://arxiv.org/html/2610.14005#A3.T12), the second to[Figure3](https://arxiv.org/html/2610.14005#S7.F3)\(right\), the third to the rankings in Appendix[E](https://arxiv.org/html/2610.14005#A5), and the last to the supra window in[Table4](https://arxiv.org/html/2610.14005#S7.T4)\.The correlation betweenK​t^\|\\widehat\{Kt\}^\{\|\}and accuracy is significant for all three reference machines\. It is also significant at the 5% level for each of the twelve examinees taken separately, with values from−0\.37\-0\.37to−0\.75\-0\.75under Qwen3\-14B\. Ability correlates with ECI under all three machines, with Pearson coefficients between\+0\.84\+0\.84and\+0\.93\+0\.93andp<0\.01p<0\.01in each case, and the three machines rank the examinees in nearly the same order\.

Gemini 3\.6 Flash is also the composer, verifier, typer and judge, which could favour it as an examinee in several ways\. Unlike the reference machines, which were set apart and never included as proper examinees \(see Appendix[E](https://arxiv.org/html/2610.14005#A5)\), we kept Gemini 3\.6 Flash in the pool because we wanted to observe the effect, and also because the extreme decision would have been to exclude all models of the same family\. Instead we analyse its effect here\. As judge, it might mark its own answers leniently\. Comparing the judge’s verdicts with a mechanical comparison of the five answer terms against the intended ones, over the 153 attempts per examinee on the items certified under Qwen3\-14B, the judge accepted one answer from Gemini 3\.6 Flash that the mechanical check rejects, against between zero and three for each of the other examinees, and it never rejected a correct answer\. As generator, it might share habits with the sequences it wrote\. Against a line fitted between ability and ECI, Gemini 3\.6 Flash lies 12 bits above, but GPT\-5\.6\-sol and Gemini 3\.8 Flash, which take no part in the pipeline, lie 11 and 9 bits above, and the residuals of the ten examinees range from−26\-26to\+21\+21bits\. Gemini 3\.8 Flash also scores above it, at 143 bits against 140\. Any advantage is therefore small, and within the spread of the other examinees\.

In the comparison of measures in Tables[8](https://arxiv.org/html/2610.14005#A1.T8),[10](https://arxiv.org/html/2610.14005#A2.T10)and[12](https://arxiv.org/html/2610.14005#A3.T12),K​t^\|\\widehat\{Kt\}^\{\|\}has the highest correlation with accuracy\. Under Qwen3\-14B, a paired bootstrap over items does not separate it from the other complexity measures\. Its difference from the unconditionalK^ℳ𝒱\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\},−0\.79\-0\.79against−0\.69\-0\.69, has an interval of\[−0\.23,\+0\.01\]\[\-0\.23,\+0\.01\], and its difference fromc\(x1:n\)c\(x\_\{1:n\}\)one of\[−0\.26,\+0\.02\]\[\-0\.26,\+0\.02\]\. From the conditionalK^\|\\widehat\{K\}^\{\|\}andK​c^\\widehat\{Kc\}it differs by only0\.010\.01and0\.020\.02\. The one significant difference is withlog2⁡τ\\log\_\{2\}\\taualone,−0\.79\-0\.79against−0\.65\-0\.65, a difference of−0\.14\-0\.14with interval\[−0\.29,−0\.01\]\[\-0\.29,\-0\.01\]\. This is natural, as many of these metrics express very similar things, and shows that the logarithm of the inference steps does not have an important effect yet, at this level of complexity, but this may change as more complex items and billions of reasoning tokens are used in the future\.

One might expect the time termlog2⁡τ\\log\_\{2\}\\tauto matter more at the upper end, where the computations are longer\. But here is where the other term inK​t\|Kt^\{\|\}is higher too\. Under Qwen3\-14B, adding it toK^\|\\widehat\{K\}^\{\|\}changes the correlation with accuracy by at most 0\.02 in the upper half of the items, and by up to 0\.08 in the lower half, where it is the only term that orders the items continued with no description\. The logarithm grows slowly, so that even a million tokens would add only 20 bits, and the largest annotation budget, 12000 tokens, caps it at about 13\.6 bits\. UsingK\|K^\{\|\}rather thanK​t\|Kt^\{\|\}would be perfectly justified too, especially for this battery, and at these levels of inference tokens\. It was not our goal to exclude other metrics, especially those based on Kolmogorov complexity or those that emphasise the benefits of costing using the reference machine\. This is what we are advocating for\. The research question was about checking whetherK​t\|Kt^\{\|\}works and it does\. Also, we think that in agentic scenarios, the reasoning tokens compared to the size of the hints may grow considerably, and inference time may become much more relevant\. Removing it in this first test and formulation, because it just adds little correlation, would be against the theoretical justification \(such as\([Levin, 1973](https://arxiv.org/html/2610.14005#bib.bib65);[Achille and Soatto, 2026](https://arxiv.org/html/2610.14005#bib.bib70)\)\) and would remove a potential key signal in the future\.

The results about the supra window should also be taken with care\. Its lower end in[Table4](https://arxiv.org/html/2610.14005#S7.T4)is where the fitted curve crosses one third, and the window above it holds few items, 10 above 190 bits under Qwen3\-14B\. With this number, an exact interval cannot exclude a rate above one third unless no item is solved\. Doubling the size of the battery would likely give much narrower intervals\. Still, on these 10 items gpt5sol answers 27% of its attempts correctly and solves 2 on a majority of attempts, with an interval of\[3%,56%\]\[3\\%,56\\%\]\. But this is the most powerful model\. The other five strongest examinees solve between 2 of 14 and 4 of 11 items in their own windows\. Seven of the ten items above 190 bits are solved on a majority of attempts by none of these six\. For the four weaker API examinees, whose windows hold 20 to 27 items, the intervals do exclude a rate above one third, with upper ends between 22% and 32%\. So the position and length of each window are uncertain, but a weak model such as Qwen3\-14B can certify items that the strongest examinees mostly fail\.

The logistic fits add2​N2Npseudo\-observations at\(0,1\)\(0,1\), twice as many as there are items, which anchors each curve at full accuracy for an item of zero bits\. We do this as we would expect all models to succeed at items of difficulty close to zero and the results already point in that direction\. In any case, the results for abilities or infra/supra windows are not sensitive to this choice\. Refitting under Qwen3\-14B with00,N/2N/2,NNor4​N4Nsuch points leaves the order of the twelve examinees unchanged \(Spearman\+1\.00\+1\.00against the fit with2​N2N, and\+0\.99\+0\.99with4​N4N\), moves each ability by at most 6 bits, and keeps the Pearson correlation with ECI between\+0\.87\+0\.87and\+0\.88\+0\.88\. Without any prior, the supra windows of the six strongest examinees start 2 to 8 bits later, so that of gpt5sol runs from 197 bits instead of 190\.

### F\.1How many rivals are enough

The unquestionability check composes six rivals at each rung of the ladder\. Here we measure how close that number is to saturation, that is, how much the verdicts would change with fewer\.

A*rung*is a prefix lengthLLin the ladderΛ\\Lambdaat which rivals are composed\. A*rival*atLLis a descriptionqqthat reproduces the prefixx1:Lx\_\{1:L\}and departs from the sequence at termL\+1L\+1, as verified by𝒱\\mathcal\{V\}\. There are two kinds\. A*composed rival*is written by the composer, which is asked for six at each rung \(a few fail verification, so some rungs have fewer\)\. A*structural rival*is not composed but constructed\. It states theLLterms of the prefix and then departs from them, in one of three ways, by repeating the prefix from its first term, by repeating its last term, or by continuing with a constant\. The structural rivals exist at every rung\. A rival*blocks*the item atLLif it costs no more than the least expensive description of the whole sequence plus the tolerance,cℳ​\(q\)≤K^\+δc\_\{\\mathcal\{M\}\}\(q\)\\leq\\widehat\{K\}\+\\delta\. The item is unquestionable atLLif no rival blocks there, andLuL\_\{u\}is the first rung at which this holds\.

For each reference machine we took the 101 valid items that have composed rivals at some rung, and recomputedLuL\_\{u\}using onlykkof the composed rivals at each rung, keeping all the structural rivals, fork=1,…,5k=1,\\dots,5\. Fewer rivals can only remove blockers, so the only possible change is that a rung beforeLuL\_\{u\}is no longer blocked, which makesLuL\_\{u\}shorter or makes a questionable item unquestionable\. We report two measures\. The first is the number of items for which at least one choice ofkkrivals would changeLuL\_\{u\}, a worst case\. The second is the expected number of items whoseLuL\_\{u\}changes when thekkrivals at each rung are drawn at random from those composed\. It is computed exactly and not by simulation\. At a rung withnncomposed rivals, of whichbbblock, and no blocking structural rival, the probability thatkkrandom rivals include no blocker is\(n−bk\)/\(nk\)\\binom\{n\-b\}\{k\}/\\binom\{n\}\{k\}\. Being an expectation, this second measure need not be a whole number\.

Table 18:How the unquestionability verdicts depend on the number of composed rivals, over the 101 valid items with composed rivals at some rung\. The last two blocks give the number of items whoseLuL\_\{u\}would change with onlykkof the composed rivals, for some choice of thekk\(worst case\) and in expectation over a random choice\.Table[18](https://arxiv.org/html/2610.14005#A6.T18)shows that six composed rivals are close to saturation\. Under Qwen3\-14B and Phi\-4\-reasoning, no choice of four of the six would change any verdict, and under Qwen3\-8B at most two items would change with four and one with five\. Two facts explain this\. First, the structural rivals do most of the work\. Whenever a composed rival blocks, a structural rival blocks as well, except at one to three rungs per machine\. Second, where composed rivals block they overlap, with a median of three or four of the six blocking, so that dropping some of them rarely leaves a rung unblocked\. More rivals of the same kind are therefore unlikely to change any verdict\.

This does not mean that the procedure finds every rival, but the level of compression is what matter most\. We could also manually changeδ\\delta, if would like to reduce further reduced false positives\.

## Appendix GWorked Examples

We presented the pipeline succinctly in the paper \([Section5](https://arxiv.org/html/2610.14005#S5)\)\. The next section describes it in full detail\. In between, it is very helpful to see how a few examples evolved during the pipeline, with actual descriptions and costs, as a trace\.

### G\.1Item 0123:K​t\|^=41\.8\\widehat\{Kt^\{\|\}\}=41\.8

We follow our first item,0123, through every stage under Qwen3\-14B, and then summarise what happened to it under the other two reference machines\.

#### Generation and validation\.

The generator asked for the cell\(r0,t0\)=\(3,0\)\(r\_\{0\},t\_\{0\}\)=\(3,0\), three retrievals and no transformation, with the theme*michelin star tier levels*from the domain*cuisine*\. The composer returned theseed

> alternating between michelin star tiers, basic taste modalities, and days of the week in order

and the sequence it produces, whose first terms are

> 1 star, sweet, monday, 2 stars, sour, tuesday, 3 stars, salty, wednesday, 1 star, bitter, thursday, 2 stars, umami, friday, 3 stars, sweet, saturday, …

The three tables have periods 3, 5 and 7, so the sequence repeats only after 105 groups\. The item was generated withN=70N=70terms\. The verifier ran the seed cold times and reproduced all 70 terms each time, so the item is valid\.

The composer then wrote nine equivalents, of which seven reproduced the sequence on verification\. They range from thebareversion

> Alternate items from Michelin star tiers, basic tastes, and lowercase days of the week\.

to pseudocode such as “A, B, C = Michelin star tiers, basic tastes, lowercase days of the week; term 3k\+1 is A\[k%3\], 3k\+2 is B\[k%5\], 3k\+3 is C\[k%7\]”\. It also wrote 21 rivals at the ladderΛ=\{5,13,47,60\}\\Lambda=\\\{5,13,47,60\\\}\. AtL=5L=5they depart at term 6, for example by running the days backwards \(giving*sunday*\) or by using the months \(giving*march*\)\. AtL=13L=13the prefix ends just before the fifth taste, and all six rivals depart at exactly that term\. One uses the four classic Western tastes and gives*sweet*, one the six Ayurvedic tastes and gives*pungent*, and the others give*savory*,*spicy*,*fatty*and*astringent*\.

#### Unquestionability under Qwen3\-14B\.

The least expensive equivalent is the bare statement above, atK^ℳ𝒱​\(x\)=131\.5\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=131\.5bits, and so it isp▼p^\{\\blacktriangledown\}\. With the clause constantδ=20\.6\\delta=20\.6bits, a rival blocks the item atLLif it costs at most 152\.1 bits\.

AtL=5L=5the item is blocked by the three structural rivals\. Repeating the five terms shown costs 138\.9 bits, repeating the last term costs 148\.2 and continuing with a constant costs 135\.7, all inside the bar\. AtL=13L=13the same three rivals cost between 165\.0 and 168\.7 bits, and the simplest rival of any kind is the constant one at 165\.0\. None of the six composed rivals comes closer\. The item is therefore unquestionable atLu=13L\_\{u\}=13, with 13\.0 bits to spare\.

#### Measuredr^\\hat\{r\}andt^\\hat\{t\}\.

The typer readp▼p^\{\\blacktriangledown\}five times and split it identically each time into three retrievals \(*Michelin star tiers*,*basic tastes*and*days of the week*\), one operation \(*lowercase*\) and control \(*alternate items from*and the joining punctuation\)\. This givesr^=3\\hat\{r\}=3andt^=1\\hat\{t\}=1with no spread, against the\(3,0\)\(3,0\)that was asked for\. The one operation is the conversion to lowercase, which the seed left implicit\.

#### Certification under Qwen3\-14B\.

The annotator showed Qwen3\-14B the first 13 terms and asked for the next 20\. Table[19](https://arxiv.org/html/2610.14005#A7.T19)lists the candidates it ran\. Each is costed conditionally on the prefix, so a hint only has to supply what the 13 terms do not already show\. The prefix shows the three star tiers and four days in full, but only four of the tastes\.

Table 19:The candidates run for0123under Qwen3\-14B, with their conditional costcℳ\(p∣x1:13\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:13\}\), the tokens used, and how many of the 20 asked terms came out correct and in order\.The empty description failed, and so did*basic tastes*, and both failed in the same way\. Each time Qwen3\-14B continued with*sweet, friday, 3 stars, sour, saturday*, getting the stars and the days right and cycling back to*sweet*after four tastes, since four is all the prefix shows\.*5 basic tastes*costs less than one bit more and succeeded at once, because the number fixes a fifth taste, umami\. It is the winning conditional description, withK^\|=31\.6\\widehat\{K\}^\{\|\}=31\.6bits andτ=1184\\tau=1184tokens\.

Two less expensive hints,*days of the week*at 26\.3 bits and*general knowledge*at 29\.9, were costed but not run, given the limited number of attempts of the algorithm\. Neither says anything about the tastes, so neither is likely to have certified, but the case shows why the value reported forK^\|\\widehat\{K\}^\{\|\}is an upper bound\. The unconditional equivalents, from 119\.4 bits upwards, were not run either\.

#### The item’s measures\.

Table[20](https://arxiv.org/html/2610.14005#A7.T20)gives the measures for the item\. The difference between the unconditional and conditional costs, 99\.9 bits, is the part of the description that the prefix makes unnecessary\. ItsK​t^\|\\widehat\{Kt\}^\{\|\}of 41\.8 bits places it well below the median of the 51 certified items, which is 101 bits\.

Table 20:The measures for0123under Qwen3\-14B, in bits\.K^ℳ𝒱​\(x\)\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\), the whole sequence, unconditional131\.5cℳ\(x1:13\)c\_\{\\mathcal\{M\}\}\(x\_\{1:13\}\), the prefix itself97\.7K^\|\\widehat\{K\}^\{\|\}, the winning hint given the prefix31\.6log2⁡τ\\log\_\{2\}\\tau, withτ=1184\\tau=1184tokens10\.2K​t^\|=K^\|\+log2⁡τ\\widehat\{Kt\}^\{\|\}=\\widehat\{K\}^\{\|\}\+\\log\_\{2\}\\tau41\.8K​c^\|=K^\|\+cℳ\(x1:13\)\\widehat\{Kc\}^\{\|\}=\\widehat\{K\}^\{\|\}\+c\_\{\\mathcal\{M\}\}\(x\_\{1:13\}\)129\.3
#### The examinees\.

Each examinee saw the same 13 terms and was asked for the next five,*umami, friday, 3 stars, sweet, saturday*, three times\. Table[21](https://arxiv.org/html/2610.14005#A7.T21)gives the results under full marking\. Fourteen of the 36 attempts are correct, and four examinees answer correctly on a majority of their attempts\.

Table 21:Attempts answered correctly on0123, out of three, with a typical wrong answer where there is one\.The most common wrong answer,*sweet, friday, 3 stars, sour, saturday*, keeps the stars and the days right and cycles four tastes, returning to*sweet*where*umami*belongs\. It is the continuation of the four\-taste rival that the composer wrote atL=13L=13, and the one Qwen3\-14B itself gave when it had no hint or a hint without the number\. Gemini 3\.5 Flash\-Lite and Gemini 3\.1 Flash\-Lite sometimes answer out of step, starting from the wrong position in the cycle\. The item therefore separates the examinees on one piece of knowledge, whether there is a fifth basic taste, which is the piece its hint supplies\.

#### Under the other two reference machines\.

Qwen3\-8B also finds the item unquestionable atLu=13L\_\{u\}=13, withK^ℳ𝒱​\(x\)=127\.6\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=127\.6bits\. Its cheapest rival at 13 costs 47\.0 bits more than that, against aδ\\deltaof 22\.7, which leaves 24\.3 bits of slack\. It certified the item as well, but with a longer hint,*Interleave 1\-3 stars, five basic tastes, and days of the week*, at 63\.6 bits and 2528 tokens, which givesK​t^\|=74\.9\\widehat\{Kt\}^\{\|\}=74\.9bits\. Like Qwen3\-14B, it failed with the empty description and with*Michelin star tiers*alone\.

Phi\-4\-reasoning needed a longer prefix\. AtL=13L=13its structural rivals cost 149\.5 and 151\.9 bits, inside its bar of 156\.6, so the item was blocked\. ByL=26L=26the structural rivals were outside the bar, but no rivals had been composed at 26, so the next length that could be tested was 47, where the item became unquestionable\. Certification then failed\. The seed and one equivalent ran to the limit of 1500 tokens without producing a correct term, and a second equivalent and three hints stopped within the limit, also with none correct\. Only the listing of the continuation succeeded\. That is a fallback and not a certification, so the item is not in Phi\-4\-reasoning’s battery\.

### G\.2Item 0049:K​t\|^=50\.3\\widehat\{Kt^\{\|\}\}=50\.3

Item0123is almost pure retrieval\. Here we follow0049, which has one retrieval and three transformations, again under Qwen3\-14B first and then under the other two machines\. It too certified with a short hint\.

#### Generation and validation\.

The generator asked for the cell\(r0,t0\)=\(1,3\)\(r\_\{0\},t\_\{0\}\)=\(1,3\)with the theme*months of the year*from the domain*chronology*\. The composer returned the seed

> from n=1; take the nth month name in order, reverse it, keep the first 3 letters, and append n

which gives

> yra1, yra2, hcr3, lir4, yam5, enu6, ylu7, tsu8, reb9, reb10, reb11, reb12, yra13, yra14, hcr15, …

Keeping three letters hides part of each name\. January and February both give*yra*, and September to December all give*reb*, so the letters alone do not identify the month and only the index keeps the terms distinct\. The item was generated withN=71N=71terms, and the verifier reproduced all of them\.

The composer wrote nine equivalents and all nine reproduced the sequence, from*Reverses each month’s name, takes the first 3 letters, and appends the term index n\.*to pseudocode such asa\_n = rev\(lower\(nth English month\)\)\[1\.\.3\] \+ n\. It also wrote 16 rivals at the ladderΛ=\{5,13,23\}\\Lambda=\\\{5,13,23\\\}\. The five rivals atL=5L=5all depart at term 6, June, and each keeps the rule and adds an exception\. One uses the whole reversed name for months of fewer than five letters and gives*enuj6*\. Another leaves months ending in*e*unreversed and gives*jun6*\. A third shifts every letter forward whennnis a multiple of 6 and gives*fov6*\. The rivals atL=13L=13change which month is taken after the first year, for example the\(n−12\)2\(n\-12\)^\{2\}\-th, which gives*lir14*\.

#### Unquestionability under Qwen3\-14B\.

The least expensive description of the whole sequence is the first equivalent above, atK^ℳ𝒱​\(x\)=128\.3\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=128\.3bits\. The bare statement costs 150\.1 bits, so here it is notp▼p^\{\\blacktriangledown\}\. Withδ=20\.6\\delta=20\.6bits, a rival blocks the item if it costs at most 148\.9 bits\.

The item is unquestionable at the first length tried,Lu=5L\_\{u\}=5\. A structural rival has to state the five terms, and strings such as*yra1, yra2, hcr3, lir4, yam5*are expensive to state\. Repeating them costs 162\.9 bits, repeating the last one 162\.2 and continuing with a constant 159\.5, all outside the bar, and each composed rival costs more again, because its exception has to be stated as well\. The slack is 10\.5 bits\.

#### Measuredr^\\hat\{r\}andt^\\hat\{t\}\.

The typer readp▼p^\{\\blacktriangledown\}five times and gave the same answer each time\. It found one retrieval, the months, and three operations, reversing the name, keeping the first three letters and appending the index\. This givesr^=1\\hat\{r\}=1andt^=3\\hat\{t\}=3with no spread, exactly the cell that was asked for\.

#### Certification under Qwen3\-14B\.

The annotator showed Qwen3\-14B the first five terms and asked for the next 20\. Table[22](https://arxiv.org/html/2610.14005#A7.T22)lists the candidates it ran\.

Table 22:The candidates run for0049under Qwen3\-14B, with their conditional costcℳ\(p∣x1:5\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:5\}\), the tokens used, and how many of the 20 asked terms came out correct and in order\.With no description, Qwen3\-14B did not find the months at all\. It treated the letters as numbers and looked for differences between them, as in*l→\\toy \(\+13\), i→\\toa \(−8\-8\)*\. Told*nth month*, it recognised the months and worked out that the letters come from the end of each name, but made mistakes and got only four of the 20 terms right, even with 6000 tokens\.*Month names reversed*costs one bit more and succeeded within the first budget, because it names the reversal, the step the model could not find by itself\. The prefix already shows the truncation to three letters and the index, so the hint does not need to mention them\. It is the winning conditional description, withK^\|=39\.5\\widehat\{K\}^\{\|\}=39\.5bits andτ=1792\\tau=1792tokens\.

Four cheaper hints were costed but not run\. Two of them,*months*at 23\.2 bits and*months of the year*at 26\.4, say no more than*nth month*, which failed\. The other two,*Months spelled backwards*at 37\.2 bits and*Months reversed*at 38\.0, say almost exactly what the winning hint says and would quite possibly have certified\. This is a concrete case of the upper bound discussed in Appendix[L](https://arxiv.org/html/2610.14005#A12)\. The value ofK​t^\|\\widehat\{Kt\}^\{\|\}reported here is likely to be a few bits too high\.

#### The item’s measures\.

Table 23:The measures for0049under Qwen3\-14B, in bits\.K^ℳ𝒱​\(x\)\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\), the whole sequence, unconditional128\.3cℳ\(x1:5\)c\_\{\\mathcal\{M\}\}\(x\_\{1:5\}\), the prefix itself100\.6K^\|\\widehat\{K\}^\{\|\}, the winning hint given the prefix39\.5log2⁡τ\\log\_\{2\}\\tau, withτ=1792\\tau=1792tokens10\.8K​t^\|=K^\|\+log2⁡τ\\widehat\{Kt\}^\{\|\}=\\widehat\{K\}^\{\|\}\+\\log\_\{2\}\\tau50\.3K​c^\|=K^\|\+cℳ\(x1:5\)\\widehat\{Kc\}^\{\|\}=\\widehat\{K\}^\{\|\}\+c\_\{\\mathcal\{M\}\}\(x\_\{1:5\}\)140\.1Table[23](https://arxiv.org/html/2610.14005#A7.T23)gives the measures\. The prefix makes 88\.8 bits of the description unnecessary\. The prefix is also expensive in itself\. Its five terms cost 100\.6 bits, about as much as the thirteen terms of the first example \(Appendix[G\.1](https://arxiv.org/html/2610.14005#A7.SS1)\), because strings such as*hcr3*are unlikely text\. WithK​t^\|=50\.3\\widehat\{Kt\}^\{\|\}=50\.3bits the item is still below the median of the certified items, 101 bits\.

#### The examinees\.

Each examinee saw the five terms and was asked for*enu6, ylu7, tsu8, reb9, reb10*\. Twenty\-one of the 36 attempts are correct, and seven examinees answer correctly on a majority of their attempts\.

Table 24:Attempts answered correctly on0049, out of three, with a typical wrong answer where there is one\.The wrong answers again follow the rivals\. The most common one,*yra6, yra7, hcr8, lir9, yam10*, repeats the five letter groups shown with new indices, which is the continuation of the structural rival that repeats the prefix\. This is the rival the item had to exclude to be unquestionable\. Gemini 3\.1 Flash\-Lite takes the last three letters of each month and writes them forwards\. Its first term,*une6*, is also the first term of one of the composed rivals\. The six strongest examinees solve the item on every attempt\.

#### Under the other two reference machines\.

Both other machines also find the item unquestionable atLu=5L\_\{u\}=5, Qwen3\-8B withK^ℳ𝒱​\(x\)=114\.8\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=114\.8bits and 19\.9 bits of slack, and Phi\-4\-reasoning with 110\.7 bits and 30\.8 bits of slack\. They differ from Qwen3\-14B in certification\. For Qwen3\-8B every hint failed, including*first 3 letters of reversed months*, and only the full equivalent*Reverses each month’s name, takes the first 3 letters, and appends the term index n\.*certified, at 102\.1 bits and 2688 tokens\. For Phi\-4\-reasoning every hint failed as well, one equivalent reached 10 of the 20 terms, and another certified, at 117\.0 bits and 1728 tokens\. The item therefore hasK​t^\|=50\.3\\widehat\{Kt\}^\{\|\}=50\.3bits under Qwen3\-14B, 113\.5 under Qwen3\-8B and 127\.7 under Phi\-4\-reasoning\. The three machines agree that five terms are enough to fix the continuation, but only Qwen3\-14B can continue it from a three\-word hint, and the other two need the whole rule\. This is the kind of difference between reference machines that Figures 4, 8 and 12 show across the battery\.

### G\.3Item 0003:K​t\|^=200\.7\\widehat\{Kt^\{\|\}\}=200\.7

The two previous items are at the easy end of the scale\. Here we follow0003, which is certified under Qwen3\-14B atK​t^\|=200\.7\\widehat\{Kt\}^\{\|\}=200\.7bits, inside the supra window of every examinee in Table 4, and which only two of the twelve examinees solve\.

#### Generation and validation\.

The generator asked for the cell\(r0,t0\)=\(1,4\)\(r\_\{0\},t\_\{0\}\)=\(1,4\)with the theme*bovine milk fat percentages*from the domain*farming*\. The composer returned the seed

> A ledger starts with fat percentages \[1, 3\]\. Pop the lowest, emit its element symbol and ’ pct fat’\. Add double the percentage modulo 118 as a new debt, filter out 0, and sort ascending\.

The theme appears only as decoration, in the suffix*pct fat*\. What the rule describes is two numbers, starting at 1 and 3\. At each step the smaller one is printed as the symbol of the chemical element with that atomic number, and is then replaced by its double modulo 118, the number of elements in the periodic table\. The first terms are

> h pct fat, he pct fat, li pct fat, be pct fat, c pct fat, o pct fat, mg pct fat, s pct fat, …

with atomic numbers 1, 2, 3, 4, 6, 8, 12, 16, which is the two doubling sequences from 1 and from 3 merged in order\. The sequence looks like the periodic table until it skips boron\. At term 13 the doubling of 64 wraps round to 10, neon, and from there the atomic numbers look erratic\. By term 37 the two numbers have fallen into the same cycle, eight steps apart, so every block of eight atomic numbers is printed twice in succession\. The item was generated withN=72N=72terms\.

The composer wrote 17 equivalents, but only six reproduced the sequence when the verifier ran them\. The queue is easy to state loosely and hard to state exactly\. It also wrote 15 rivals at the ladderΛ=\{5,13,60\}\\Lambda=\\\{5,13,60\\\}\. AtL=5L=5they read the atomic numbers 1, 2, 3, 4, 6 in other ways\. The divisors of 12 give magnesium next, the divisors of 36 fluorine, the integers not divisible by 5 nitrogen, and the integers ordered by Euler’s totient boron\. AtL=13L=13they keep the queue and change the doubling after the wrap, and atL=60L=60they change what happens near the top of the table\.

#### Unquestionability under Qwen3\-14B\.

The least expensive description of the whole sequence is an equivalent,*Start with 1 and 3\. Output lowercase chemical symbol for smallest n plus ’ pct fat’; replace n with 2n mod 118 \(discarding 0\); repeat\.*, atK^ℳ𝒱​\(x\)=272\.1\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=272\.1bits, so the bar for a rival is 292\.7 bits\.

AtL=5L=5eight rivals come inside it, the four number\-theoretic readings among them\. AtL=13L=13none of the composed rivals does, but three structural ones still cost less than the bar\. FromL=27L=27on the structural rivals cost more than the bar as well, but no rivals were composed at 27, so the next length that could be tested was 60\. There the simplest rival is a composed one at 373\.5 bits, and the item is unquestionable atLu=60L\_\{u\}=60with 80\.8 bits to spare\. The prefix of 60 terms is therefore set by the ladder and not by the item, which would probably have been unquestionable from 27 terms\. This matters for the examinees, as we will see\.

#### Measuredr^\\hat\{r\}andt^\\hat\{t\}\.

The typer readp▼p^\{\\blacktriangledown\}five times and gaver^=1\\hat\{r\}=1andt^=5\\hat\{t\}=5each time\. The one retrieval is the chemical elements, and the operations are those of the queue and the formatting\. This is one more operation than the four the cell asked for\.

#### Certification under Qwen3\-14B\.

Qwen3\-14B was shown the first 60 terms and asked for the nextku=min⁡\(20,72−60\)=12k\_\{u\}=\\min\(20,72\-60\)=12\. Table[25](https://arxiv.org/html/2610.14005#A7.T25)lists the candidates it ran\.

Table 25:The candidates run for0003under Qwen3\-14B, with their conditional costcℳ\(p∣x1:60\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:60\}\), the tokens used, and how many of the 12 asked terms came out correct and in order\.Every hint failed\. With the theme as a hint, Qwen3\-14B found that the atomic numbers double, writing*O \(8\), S \(16\), Ge \(32\), Gd \(64\)*, but it did not find the second number or the modulus\. A loose equivalent,*Settles milk fat debts in order by element symbol, doubling and wrapping percentages to queue new debts*, reached eight of the twelve terms\. Only the full algorithm certified,

> L = \[1, 3\]; loop: x = min\(L\); L\.remove\(x\); L\.append\(2\*x % 118\); output lowercase periodic symbol x \+ ’ pct fat’\.

and even then it failed with 3000 and with 6000 tokens and succeeded with 12000, using 10464\. Five simpler hints, such as*Periodic table*and*Element symbols*, were costed but not run\. Given that the other element hints failed, they are unlikely to have certified\.

#### The item’s measures\.

Table 26:The measures for0003under Qwen3\-14B, in bits\.K^ℳ𝒱​\(x\)\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\), the whole sequence, unconditional272\.1cℳ\(x1:60\)c\_\{\\mathcal\{M\}\}\(x\_\{1:60\}\), the prefix itself463\.1K^\|\\widehat\{K\}^\{\|\}, the winning description given the prefix187\.4log2⁡τ\\log\_\{2\}\\tau, withτ=10464\\tau=10464tokens13\.4K​t^\|=K^\|\+log2⁡τ\\widehat\{Kt\}^\{\|\}=\\widehat\{K\}^\{\|\}\+\\log\_\{2\}\\tau200\.7K​c^\|=K^\|\+cℳ\(x1:60\)\\widehat\{Kc\}^\{\|\}=\\widehat\{K\}^\{\|\}\+c\_\{\\mathcal\{M\}\}\(x\_\{1:60\}\)650\.5Table[26](https://arxiv.org/html/2610.14005#A7.T26)gives the measures\. Here the prefix saves only 84\.7 bits of a 272\.1\-bit description, because the rule has to be stated in full even after 60 terms\. The computation also counts for more than in the two previous items\. Herelog2⁡τ\\log\_\{2\}\\tauadds 13\.4 bits, against about 10 for them\.

#### The examinees\.

The examinees were shown the same 60 terms and asked for the next five,*cn, sg, pu, yb, ti*, with atomic numbers 112, 106, 94, 70 and 22\. These five begin the second copy of the last block of eight, so they can be found by running the rule, or by noticing that every block appears twice\.

Table 27:Attempts answered correctly on0003, out of three, and the kind of answer given\. Atomic numbers are in parentheses\.The answers sort into three levels of understanding \(Table[27](https://arxiv.org/html/2610.14005#A7.T27)\)\. The weakest reading continues the periodic table from the last element shown, cerium, giving the elements 59 to 63\. The middle one, given by gpt5sol on every attempt and by several of the strongest Gemini and Claude models, doubles the last atomic number modulo 118 and keeps doubling,58→116→114→110→102→8658\\to 116\\to 114\\to 110\\to 102\\to 86\. These examinees found the doubling and the modulus, which is most of the rule, but treated the sequence as a single number and missed the second one\. Only Claude Sonnet 5\* and Claude Opus 5\* give the correct continuation on a majority of their attempts\.

One attempt is marked in the results as correct although it is not\. The first of Gemini 3\.8 Flash’s three attempts gives the doubled answer, identical to its other two, but the judge scored it five out of five\. Counting it as wrong, 6 of the 36 attempts are correct\.

The ladder affects this item in a second way\. Because the examinees see 60 terms and not the 27 that would probably have sufficed, the repetition of the blocks is visible, and an examinee can answer the five terms asked from the repetition alone without finding the rule\. The reference machine was asked for twelve terms, and the last four of them start a new block that only the rule predicts\. So the item as examined is likely to be somewhat easier than itsK​t^\|\\widehat\{Kt\}^\{\|\}says\.

#### Under the other two reference machines\.

Both other machines also find the item unquestionable atLu=60L\_\{u\}=60, with 103\.4 bits to spare under Qwen3\-8B and 53\.6 under Phi\-4\-reasoning\. Neither certifies it\. Under both, the hints, the loose equivalent and the full algorithm mostly produced the first eight of the twelve terms and then went wrong, which is what an examinee gets from the repetition of the blocks without the rule\. This includes Phi\-4\-reasoning with the hint*Double atomic number modulo 118*, and both machines with the full algorithm\. Their budgets did not reach the 10464 tokens that Qwen3\-14B needed\. Only the listing of the continuation succeeded, which is a fallback and not a certification, so the item is in the battery of Qwen3\-14B alone\.

### G\.4Item 0084:K​t\|^=302\.9\\widehat\{Kt^\{\|\}\}=302\.9

The last item,0084, is one of the two hardest certified items that no examinee answers correctly on any attempt\. This one is only certified under Qwen3\-14B atK​t^\|=302\.9\\widehat\{Kt\}^\{\|\}=302\.9bits\.

#### Generation and validation\.

The generator asked for the cell\(r0,t0\)=\(1,7\)\(r\_\{0\},t\_\{0\}\)=\(1,7\)with the theme*snooker ball point values*from the domain*games*\. The composer returned the seed

> For term n from 1, take the n\-th snooker ball in point\-value order \(1: red, 2: yellow, 3: green, 4: brown, 5: blue, 6: pink, 7: black, cycling every 7\)\. Multiply n by the ball’s point value, square the result, take its digit sum, add n to this sum, reverse the digits of the sum, divide by 2 if even, and format as the ball name and final number\.

Each term has two parts\. The ball cycles through the seven colours and is easy to continue\. The number needs six steps\. Forn=4n=4the ball is brown, worth 4, and4×4=164\\times 4=16,162=25616^\{2\}=256, whose digit sum is 13, plus 4 gives 17, reversed 71, which is odd, so the term is*brown 71*\. The first terms are

> red 1, yellow 9, green 21, brown 71, blue 81, pink 21, black 41, red 81, yellow 81, green 91, brown 3, blue 6, …

The item was generated withN=72N=72terms\.

The composer wrote 17 equivalents and eleven reproduced the sequence, all of them close to pseudocode\. A loose summary,*Applies seven sequential numerical and formatting transformations to each index using snooker ball point values*, did not\. It also wrote 15 rivals at the ladderΛ=\{5,13,60\}\\Lambda=\\\{5,13,60\\\}\. AtL=5L=5five of the six keep the colours and change the arithmetic, giving numbers such as−139\-139and 152 at the sixth term\. AtL=13L=13they change the cycle, for instance to thirteen colours\. AtL=60L=60they treat the 60 terms shown as a period in one way or another, and one of them simply starts again from*red 1*\.

#### Unquestionability under Qwen3\-14B\.

The least expensive description of the whole sequence is an equivalent atK^ℳ𝒱​\(x\)=302\.7\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=302\.7bits, so the bar for a rival is 323\.3 bits\. The structural rivals come inside it at every length up to 27, where the cheapest costs 313\.9 bits\. They are outside from 38 on, but no rivals were composed at 38 or 49\. At 60 the cheapest rival is a composed one, at 328\.7 bits, only 5\.4 bits above the bar, and the item is unquestionable atLu=60L\_\{u\}=60\.

#### Measuredr^\\hat\{r\}andt^\\hat\{t\}\.

The typer found one retrieval, the snooker balls in point\-value order, on every reading\. It counted five operations on four of its five readings and six on the other, which givest^=5\.2\\hat\{t\}=5\.2with a spread of 0\.4\. The seed describes six arithmetic steps, while the cell asked for seven\.

#### Certification under Qwen3\-14B\.

Qwen3\-14B was shown the first 60 terms and asked for the next 12\. Table[28](https://arxiv.org/html/2610.14005#A7.T28)lists the ten candidates it ran, with the three hints on one line\.

Table 28:The candidates run for0084under Qwen3\-14B, with their conditional costcℳ\(p∣x1:60\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:60\}\), the tokens used, and how many of the 12 asked terms came out correct and in order\.With no description, Qwen3\-14B treated the number as a counter and continued*blue 426, pink 427, black 428*\. The three hints failed within 3000 tokens\. The loose summary failed too, and its reply contains*blue 27, pink 37, black 9, red 57*, which is the start of the wrong answer given by three of the strongest examinees\. Four equivalents in pseudocode failed, one reaching eight of the twelve terms and one failing even with 12000 tokens\. Only the seed, the most expensive candidate run, certified\.

The seed’s run ended at the budget of 6000 tokens with the answer already in its working, soτ\\tauis recorded as 6000\. Counted by hand, the answer was complete after about 5916 tokens, which would lowerK​t^\|\\widehat\{Kt\}^\{\|\}by 0\.02 bits\.

#### The item’s measures\.

Table 29:The measures for0084under Qwen3\-14B, in bits\.K^ℳ𝒱​\(x\)\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\), the whole sequence, unconditional302\.7cℳ\(x1:60\)c\_\{\\mathcal\{M\}\}\(x\_\{1:60\}\), the prefix itself455\.7K^\|\\widehat\{K\}^\{\|\}, the seed given the prefix290\.4log2⁡τ\\log\_\{2\}\\tau, withτ=6000\\tau=6000tokens12\.6K​t^\|=K^\|\+log2⁡τ\\widehat\{Kt\}^\{\|\}=\\widehat\{K\}^\{\|\}\+\\log\_\{2\}\\tau302\.9K​c^\|=K^\|\+cℳ\(x1:60\)\\widehat\{Kc\}^\{\|\}=\\widehat\{K\}^\{\|\}\+c\_\{\\mathcal\{M\}\}\(x\_\{1:60\}\)746\.1Table[29](https://arxiv.org/html/2610.14005#A7.T29)gives the measures\. The prefix saves only 12\.3 bits, so after 60 terms the rule costs almost as much as it does with none\. The colours are obvious from the prefix, but the numbers carry almost no information about how they were made\. This rule is expensive to state, andK​t^\|\\widehat\{Kt\}^\{\|\}places the item among the hardest in the battery, which the results confirm\.

#### The examinees\.

The examinees were shown the 60 terms and asked for*blue 4, pink 49, black 9, red 19, yellow 9*\. None answered correctly on any of the 36 attempts\. Nearly all of them continued the colours correctly and got every number wrong, and Table[30](https://arxiv.org/html/2610.14005#A7.T30)shows what they gave\.

Table 30:Answers to0084\. No attempt is correct\.gpt5sol, Claude Opus 5\* and Claude Fable 5\.1\* each give the same wrong continuation on at least one attempt, the one the loose summary produced under Qwen3\-14B\. Its third term,*black 9*, happens to be correct, but the first is wrong, so it scores nothing under either marking\. Gemini 3\.8 Flash repeats the numbers of terms 5 to 9, as if the sequence had period 56, and Gemini 3\.1 Flash\-Lite adds 20 each time, modulo 100\.

#### Under the other two reference machines\.

Both other machines find the item unquestionable atLu=60L\_\{u\}=60, Qwen3\-8B with 13\.4 bits to spare and Phi\-4\-reasoning with 32\.0\. Neither certifies it\. Their hints and equivalents stopped at budgets of 1500 or 3000 tokens with no correct term, well short of the 6000 that the seed needed under Qwen3\-14B\. So the item is in the battery of Qwen3\-14B alone\.

## Appendix HThe Full Pipeline

The pipeline was summarised in[Figure1](https://arxiv.org/html/2610.14005#S5.F1)\. To limit the clutter, two secondary processes are not included in the figure, but we need to include them here\. The first one is a calibration of several parameters: bits per characterγ\\gamma,c0c\_\{0\},δ\\deltaandβ\\beta, for a rough estimation ofKKused by the generator and unquestionability modules\. More precisely, these are necessary for the prefix and continuation estimation, during generation, so we know at what lengths to generate the rivals, and these parameters are part of the unquestionability condition too\. The second process is a measurement of the values of\(r,t\)\(r,t\)once the item has gone through unquestionability\. It is at this point where we have more compact programs for the item that may change the components of the items, either by cancellation \(e\.g\., two reversals being concatenated\) or because one is lost during a chain of transformations\. Occasionally,\(r^,t^\)\(\\hat\{r\},\\hat\{t\}\)can end up being larger than originally, not only lower\. Let us go through these stages in order\.

### H\.1The calibrator

The calibrator measures 379 fixed probes of a diversity of sequences of different families, under each machine that is selected, and reports several quantities, given in[Table31](https://arxiv.org/html/2610.14005#A8.T31)\. First,γ\\gamma, bits per character, which depends on the tokeniser as much as on the power of machine\. The table gives both the median and the 80th percentile, because the generator uses the percentile\. Second,δ\\deltais the cost of the shortest exception clause the machine can state, which is the shortest way of writing “and then something else”\. This is the only calibration constant in[Corollary4\.1](https://arxiv.org/html/2610.14005#S4.Thmcorollary1)\. Third,c0c\_\{0\}andβ\\betacome from one fit\. The listing of a sequence is written out at prefix lengths 5, 10, 15, 20, 25 and 30, each is costed by the machine, and a lineλ⁡\(L\)=β​L\+c0\\lambda\(L\)=\\beta L\+c\_\{0\}is fitted to those six points\.c0c\_\{0\}is the part that does not grow with the prefix, the wrapper and the framing\.β\\betais what one further term costs\.

Table 31:The calibration constants, measured on 379 fixed probes for six LLMs\. The last three were measured for robustness, but not used as reference machines\.γ\\gammais bits per character, reported as the median and the 80th percentile over the probes\.δ\\deltais the cost of the shortest exception clause\.c0c\_\{0\}andβ\\betacome from fittingλ⁡\(L\)=β​L\+c0\\lambda\(L\)=\\beta L\+c\_\{0\}to the cost of listing a sequence at prefix lengths 5 to 30\.β\\betais reported as the median over the sequence families whose fit reachesr2≥0\.75r^\{2\}\\geq 0\.75, with the range over those families in brackets and the number of families that qualify out of five\. The pooled row is what the generator read: the most conservative value of each constant over the four machines calibrated at the time \(Qwen3\-8B, gemma\-3\-4b, OLMo\-2\-7B and Llama\-3\.1\-8B\), that is, the largestγ80\\gamma\_\{80\}andδ\\delta, and the smallestc0c\_\{0\}andβ\\beta\. Itsβ\\betais the smallest value of any qualifying family, Llama\-3\.1\-8B’s on element names\.γ\\gammaγ80\\gamma\_\{80\}δ\\deltac0c\_\{0\}β\\betafitsbits/charbitsbitsbits/termreference machinesQwen3\-14B1\.572\.2120\.6121\.24\.11 \[1\.1, 10\.6\]4/5Qwen3\-8B1\.652\.1922\.7129\.611\.87 \[6\.0, 17\.7\]2/5Phi\-4\-reasoning1\.792\.1616\.3129\.89\.93 \[3\.6, 16\.3\]2/5other language models to make the estimates more robustgemma\-3\-4b\-it2\.523\.3122\.3159\.66\.08 \[3\.4, 22\.3\]4/5OLMo\-2\-7B1\.992\.4218\.7119\.210\.07 \[6\.0, 23\.5\]4/5Llama\-3\.1\-8B1\.402\.0417\.1116\.64\.88 \[1\.0, 17\.4\]5/5pooled, as used3\.31—22\.7116\.60\.98—The fit is repeated for nine families of probe sequences, so there are nine values ofβ\\betaper machine and they differ by more than an order of magnitude\. For instance, under Qwen3\-14B, counting numbers givesβ=0\.55\\beta=0\.55but interleaved two\-retrieval sequences give10\.6210\.62\. A term that is the next integer costs almost nothing to write down while term drawn from two unrelated retrievals costs a great deal\.

Only five of the nine families are used, those that have at least forty different terms, because that is what typical items look like\. Counting and weekdays are costed and reported but excluded, since aβ\\betataken from them would be an order of magnitude low\. And a family is used only if its fit reachesr2≥0\.75r^\{2\}\\geq 0\.75\. The last column of[Table31](https://arxiv.org/html/2610.14005#A8.T31)gives how many of the five qualify\. Qwen3\-8B and Phi\-4\-reasoning have only two each, so theirβ\\betais a median of two values that differ by a factor of three or four\.

Theβ\\betareported in the table is therefore a median over the qualifying families, with the range over those families in brackets\. A third value,βhigh\\beta\_\{\\mathrm\{high\}\}, is the largest of them\. It is used as a stopping rule and not as an estimate\.

With this, the generator is ready to read the last row of[Table31](https://arxiv.org/html/2610.14005#A8.T31), because the generator is independent from any reference machine in particular\. This isδgen\\delta\_\{\\mathrm\{gen\}\}= 22\.7\. On the contrary, the specificδℳ\\delta\_\{\\mathcal\{M\}\}for each reference machine \(20\.6, 22\.7, 16\.3\) are used in the unquestionable stage\.

### H\.2The Generator

The generator produces candidate items, iteratively, by growing and consulting a pool of themes of several domains, and progressively adjusting the generation to the \(r,tr,t\) grid filling the gaps\. It is meant to be called in successive turns to make grow the battery at any time\. The generator does not use the reference machineℳ\\mathcal\{M\}but a composer𝒞\\mathcal\{C\}, which can be any commodity LLM for creating candidates\.

At the start, with an empty battery, the generator asks the composer for themes, which are two to five words naming a subject with no rule and no verb\. For instance, “chess openings” and “Beaufort scale” qualify but “take the planets in order” does not\. Themes are kept in a pool that persists across runs, and grows when the existing themes have been used many times already\. Themes are clustered into domains\. With the current battery, the pool holds 350 themes over 61 domains, the most common being cuisine, sport, law, farming and games\. No domain accounts for more than a tenth of it\. See[Table39](https://arxiv.org/html/2610.14005#A16.T39)\.

Each theme is stored with three labels supplied by a judge: whether its terms are words or numbers, whether it is elementary, meaning a reader is expected to know it outright, and whether it is generative, meaning its members can be derived instead of recalled\. The pool currently holds 159 word\-termed themes against 191 number\-termed, and 161 generative against 189 not\. A theme is drawn for a slot, its use is counted, and it is then retired when it has been used more than a number of times\. This can be modified through a parameter through successive calls of the generator\.

For the themes, well\-known texts are requested explicitly \(e\.g\., a song, the beginning of a book or a speech\)\. These are clear cases of retrieval, which depend on whether the passage has been memorised by the examinee\. Only texts first published before 1900 and originally in English are admitted, which keeps the tokenised surface stable across models and avoids translations and copyright issues\. All this is indicated in theTHEMES\_PROMPT\(see[AppendixI](https://arxiv.org/html/2610.14005#A9)\)\. Separately, numeric themes are skipped with probabilityTHEME\_NUMERIC\_SKIPto avoid a high proportion of numeric sequences\.

Items are requested by cell in the grid, which is a target number of retrieval stepsr0r^\{0\}and transformation stepst0t^\{0\}, together with a theme\. As the values ofrrandttare measured at later stages of the process, the desired and actual values may diverge\. Because of that, the request is calibrated with a linear fit over the items already admitted, mapping what is asked for to what the later measures\. The correction onttis capped at two steps\. Either or both corrections can be disabled at any time \(especially for the first items\)\.

Through a specific prompt that getsrr,ttand thet​h​e​m​etheme\(seeGEN\_PROMPTin[AppendixI](https://arxiv.org/html/2610.14005#A9)\) the composer𝒞\\mathcal\{C\}returns a seed rulep0p^\{0\}in natural language together with the sequencexxit produces\. The verifier𝒱\\mathcal\{V\}, which in our experiments was the same model in a separate context, with no sight of the first generation, then runs the rule cold as a standalone prompt\. The item is valid when a second cold run reproduces the sequence the first produced\. Only at this point we have some guarantees that we have a potentially useful item\.

The item is screened for internal consistency, for overlap with items already in the battery, and for not sharing a prefix with another item\. For instance, two items whose first thirteen terms agree cannot both be unquestionable at that length\. An item that fails a screen is returned to the composer with the fault explained forrepair, and the composer tries again up to five times, before the item is discarded\. Some faults cannot be repaired and the prompt includes some of them as worked negative examples\.

The generator composes many candidates for every slot it wants to fill, but only a few may have reached this point, when they are added to the battery\. Being strict here is useful, as the generator is less expensive than other parts of the process\. At the moment, about two thirds of candidates end up being valid, and between half and three quarters of those become unquestionable, depending on the reference machine\.

The lengthNNof a sequence is fixed before anything is run\. From the intended cell\(r,t\)\(r,t\)we take the expected length of its rule in characters, and multiplying byγ\\gammagives an estimateK^\\widehat\{K\}of its cost in bits\.[Corollary4\.1](https://arxiv.org/html/2610.14005#S4.Thmcorollary1), solved for the prefix, then gives the number of terms such a rule is likely to need before it can be unquestionable,L^=\(K^\+δ−c0\)/β\\hat\{L\}=\(\\widehat\{K\}\+\\delta\-c\_\{0\}\)/\\beta\.NNis this prefix plusK\_ADMIN=12=12further terms, so that at least twelve terms are left to ask for once the prefix is shown\.NNis then kept between 28 and 72\. The lower limit leaves room for the first two rungs of the ladder, 5 and 13, with twelve terms after them\. The upper limit comes from a test of how long a sequence the verifier could run out, and it is why the ladder stops at 60\. With the constants in[Table31](https://arxiv.org/html/2610.14005#A8.T31), a rule of 40 characters givesL^=40\\hat\{L\}=40andN=52N=52, and any rule longer than about 46 characters reachesN=72N=72\.

It is only at this point that the item is added to the battery, and the generator starts preparing things for the next stages\. First, the composer is asked for eightequivalents, which are other statements of the same rule, and for sixrivalsat each prefix length on the ‘ladder’ \(a series of the numbers of terms that a retried sequentially, instead of trying one by one, which would be very expensive\)\. Formally a ladderΛ\\Lambdais a sequence of increasing natural numbers\. Two lengths are used for every item, 5 and 13, and two more are placed from the estimate in[Corollary4\.1](https://arxiv.org/html/2610.14005#S4.Thmcorollary1)\.

For each item, a line is fitted,β​L\+c0\\beta L\+c\_\{0\}unconditionally, in the frame withKℳ𝒱​\(x\)K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)andδ\\delta\.β\\betais what one further term costs underℳ\\mathcal\{M\}in this item’s vocabulary\.c0c\_\{0\}is the part that does not grow withLL, the wrapper and the framing, and at the lengths in use it is usually most of the cost\. For0030,β=3\.95\\beta=3\.95andc0=105\.7c\_\{0\}=105\.7, soλ⁡\(27\)=212\.4\\lambda\(27\)=212\.4against a limit of174\.1174\.1, and the rival is excluded by 38 bits\. The termβ​L\\beta Lalone would have left it inside\.

The condition holds of this rival whenKℳ𝒱​\(x\)\+δ<β​L\+c0K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)\+\\delta<\\beta L\+c\_\{0\}, so

L\>Kℳ𝒱​\(x\)\+δ−c0β,L\\;\>\\;\\frac\{K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)\+\\delta\-c\_\{0\}\}\{\\beta\},
This gives a lower limit on the prefix, not its exact length\. Below the limit the item is certainly questionable, because the rival that lists the prefix is cheaper than the rule\. Above it that rival is excluded, but the item still has to be checked against the composed rivals, and one of them may block it until a longer prefix\. The estimate places the last two points ofΛ\\Lambda, one just above it and one further out by a factor\. An item whose estimate falls at 24 is givenΛ=\{5,13,26,45\}\\Lambda=\\\{5,13,26,45\\\}\. As said, the ladder always contains 5 and 13\. Two more rungs are added from the estimateL^\\hat\{L\}: one just above it, atL^\+2\\hat\{L\}\+2, and one further out, at1\.8​L^\+21\.8\\,\\hat\{L\}\+2rounded, in case the estimate is too low, both capped at 60\.

So for variousL∈ΛL\\in\\Lambda, we generate rivals\. A rival atLLis a descriptionqqwith

\[𝒱\(q\)\]1:L2​@​3=x1:Land\[𝒱\(q\)\]L\+12​@​3≠xL\+1,\[\\,\\mathcal\{V\}\(q\)\\,\]^\{2@3\}\_\{1:L\}=x\_\{1:L\}\\qquad\\text\{and\}\\qquad\[\\,\\mathcal\{V\}\(q\)\\,\]^\{2@3\}\_\{L\+1\}\\neq x\_\{L\+1\},A rival reproduces the prefix exactly and then deviates from it\. Both equivalents and rivals are verified by𝒱\\mathcal\{V\}in the same way as the seed\.

As𝒱\\mathcal\{V\}is stochastic, the same description given twice need not return the same sequence, whileℳ\\mathcal\{M\}\(for any of the reference machines we are using in this paper\) decodes greedily and is run once\. Accordingly, for verification we repeat the execution again\. Namely,\[𝒱⁡\(p\)\]2​@​3\[\\,\\mathcal\{V\}\(p\)\\,\]^\{2@3\}denotes running the seed, the equivalent or rival candidates three times and taking the value returned by at least two of them, undefined if all three differ\.

The parameterβhigh\\beta\_\{\\mathrm\{high\}\}is used for one other purpose, which is to stop spending on an item\. Rivals account for about 70% of what an item costs to generate, so there is no point playing with them for a rule that could not certify at any prefix it will be shown at\. The largestβ\\betais optimistic as the cost of abandoning a rival wrongly is an item lost while the cost of not abandoning one is a few wasted calls\.

All this is convoluted but responds to optimising the generator for efficiency\. It is important, though, to clarify that this does not affect any of the definitional concepts, and without these heuristics the code would be much cleaner\. With many more resources, we could explore all prefixes up to 400 terms without making any estimates\. With the current approximation, We may lose some candidates, but the generator is much more efficient to run\.

Rivals are very important for unquestionability, and we always include the ‘structural’ rival, which is the program that states the prefix and then deviates from it as an exception\. This rival exists at every length by construction\.

### H\.3The three filters and\(r,t\)\(r,t\)measurement

An item gets itsK​t\|Kt^\{\|\}difficulty only if it passes three stages: validity, unquestionability and certification, in order\.

After generation, the rivals have been tested as severalLL, and the goal of the unquestionability step is to determine the minimumLuL\_\{u\}, the length at which the item becomes unquestionable\. Two counts of terms followLuL\_\{u\}and should not be confused\.ku=min⁡\(20,N−Lu\)k\_\{u\}=\\min\(20,\\,N\-L\_\{u\}\)is how many terms a description must produce for the third filter, whatever the item has left\.kf=5k\_\{f\}=5is how many terms an examinee is asked for\.

While the generator has executed the seed on the composer, we now need to determine if it is long enough forNN\. This involves two independent executions of the seed:

\[𝒱\(p0\)\]1:N2​@​3=\[𝒱\(p0\)\]1:N2​@​3=x1:N,\[\\,\\mathcal\{V\}\(p^\{0\}\)\\,\]^\{2@3\}\_\{1:N\}\\;=\\;\[\\,\\mathcal\{V\}\(p^\{0\}\)\\,\]^\{2@3\}\_\{1:N\}\\;=\\;x\_\{1:N\},with the second run carried to 100 terms and the appropriate prefix compared over the firstNN\. This is thevaliditystep\. No reference machine takes part\.

The next stage isunquestionability\. The item is unquestionable atLLunderℳ\\mathcal\{M\}when every rival atLLsatisfies

cℳ\(q\)\>Kℳ𝒱\(x\)\+δ,Kℳ𝒱\(x\)=min\{cℳ\(p\):\[𝒱\(p\)\]2​@​3=x1:N\},c\_\{\\mathcal\{M\}\}\(q\)\\;\>\\;K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)\+\\delta,\\qquad K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)=\\min\\\{c\_\{\\mathcal\{M\}\}\(p\):\[\\,\\mathcal\{V\}\(p\)\\,\]^\{2@3\}=x\_\{1:N\}\\\},withδ\\deltathe clause constant ofℳ\\mathcal\{M\}\. Both costs are unconditional, for the reason given in Appendix[N](https://arxiv.org/html/2610.14005#A14)\.

In this phase, equivalent programs \(and the seed\) are run onℳ\\mathcal\{M\}to produce an estimate ofK^​\(x\)\\widehat\{K\}\(x\)\. Then we find the lowestLLthat is unquestionable, butLLdoes not range over every length\. As said before, rivals are composed to diverge at a ladderΛ\\Lambdafixed before the item is judged\. Then

Lu=min⁡\{L∈Λ:cℳ​\(q\)\>Kℳ𝒱​\(x\)\+δ​for every rival at​L\},L\_\{u\}\\;=\\;\\min\\\{\\,L\\in\\Lambda\\;:\\;c\_\{\\mathcal\{M\}\}\(q\)\>K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)\+\\delta\\text\{ for every rival at \}L\\,\\\},undefined if noL∈ΛL\\in\\Lambdaqualifies, in which case the item is questionable and goes no further\. A length outsideΛ\\Lambdais never tried, because a rival must have been composed to diverge for those specific steps in the ladder\.

A rival blocks an item whencℳ​\(q\)≤K^​\(x\)\+δc\_\{\\mathcal\{M\}\}\(q\)\\leq\\widehat\{K\}\(x\)\+\\delta, soδ\\deltasets how close a rival has to come\. Phi\-4 has the narrowest of the three and its condition is therefore the strictest, while Qwen3\-8B has the widest\.

The retrieval and transformation values that were used for generation\(r0,t0\)\(r\_\{0\},t\_\{0\}\)may not match the description \(program\) that is used to characterise the item\. To remedy this, we measure them again by choosingp▼p^\{\\blacktriangledown\}, the description that achievedKℳ𝒱​\(x\)K^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\)\. If no equivalentp▼p^\{\\blacktriangledown\}exists then the bare statement \(a simplified version of the seed\) and the seed are used\. Then an LLM judge splits that description into components of three types: retrieval, meaning anything that needs something to be remembered \(the alphabet and the natural numbers are excluded\); operations, meaning anything done to the terms that needs nothing remembered; and control, which is the scaffolding\.r^\\hat\{r\}is the number of distinct retrieval components andt^\\hat\{t\}the number of distinct operations\. The call to the judge is made five times at temperature0\.40\.4and the median reported with the spread kept\. Nothing downstream depends on this new measurement\(r^,s^\)\(\\hat\{r\},\\hat\{s\}\): they place an item in the\(r,t\)\(r,t\)grid for reporting coverage and for choosing what to generate next, but no definition uses them\.

The final stage is certification, conducted by a module we call the annotator, where the evaluation and verification of programs is conditional\.

The item is certified atLuL\_\{u\}underℳ\\mathcal\{M\}when some description reproduces the continuation from it:

∃p:ℳ\(p∣x1:Lu\)=xLu\+1:Lu\+ku,ku=min\(20,N−Lu\)\.\\exists\\,p\\;:\\;\\,\\mathcal\{M\}\(p\\mid x\_\{1:L\_\{u\}\}\)\\,\\;=\\;x\_\{L\_\{u\}\+1:L\_\{u\}\+k\_\{u\}\},\\qquad k\_\{u\}=\\min\(20,\\,N\-L\_\{u\}\)\.This tries to approximateK\|=cℳ\(p▼¯∣x1:Lu\)=minpcℳ\(p∣x1:Lu\)K^\{\|\}=c\_\{\\mathcal\{M\}\}\(p^\{\\overline\{\\blacktriangledown\}\}\\mid x\_\{1:L\_\{u\}\}\)=\\min\_\{p\}c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:L\_\{u\}\}\)\.

The empty description is also valid, whenℳ\\mathcal\{M\}continues the sequence unaided\. In this case, it certifies like any other and the cost is zero\.

For items where nothing certified, the listingpλp^\{\\lambda\}supplies a value forK​c\|Kc^\{\|\}as a fallback\. It stateskuk\_\{u\}terms and is correct by construction, so it is not required to run, andkuk\_\{u\}is the length at which it is costed\. That value is an upper bound, so the item is considered not certified, and not used in the analysis\.

The two main filters, unquestionability and certification rest on different quantifiers and one is conditional and the other is not\. Unquestionability is universally quantified: one rival is enough to discard the item\. Certification is existentially quantified: as soon as one description reproduces the continuation the item is certified\. But in both cases, the more resources are put in the search the better, to be safer about unquestionability and to have a good estimate of the metrics\.

## Appendix IThe Prompts

Code prompts are fixed text with a few slots filled at call time that are used to govern the behaviour of the LLMs\. Several modules in the pipeline use prompts for the LLMs involved in the process: the composer𝒞\\mathcal\{C\}and verifier𝒱\\mathcal\{V\}in the generation and validation stages,ℳ\\mathcal\{M\}during the unquestionability and certification stages, andℰ\\mathcal\{E\}and𝒥\\mathcal\{J\}during the evaluation\. The unquestionability process only makes forward pass for costing programs usingℳ\\mathcal\{M\}, but still uses simple prompts for that\. Prompts are crucial in the behaviour of the pipeline\. Because of that, prompts are hashed with the hash being written into the run record, so the results can be traced back to the specific wording that produced it\.[Table32](https://arxiv.org/html/2610.14005#A9.T32)lists the prompt with the opening words of each\. The full prompts can be found in the code\.

Table 32:The prompts, where they are used, and an excerpt\. Slots in braces are filled at call time\.The composer is asked for themes and mechanisms in two separate calls\.THEMES\_PROMPTasks only for a subject matter, two to five words, with no rule and no verb, and requires about a quarter of each list to be well\-known texts published before 1900 in English\.GEN\_PROMPTthen asks for one sequence built around one of those themes, at a target number of retrieval and transformation steps and within a character budget\. It states one hard requirement: a single rule must fix every term with ties broken by a stated convention \(to help the composer, it includes worked examples of common faults\)\. One such fault is a rule that counts the letters of a recalled word, which asks the machine for something it cannot see \(though thinking can help with that\)\. Another is a counter with a fixed word appended, which decorates a count without adding a mechanism\. The prompt also lists the themes the battery already uses, and asks the composer to refuse a theme if it cannot work with it\.

RUN\_PROMPTgives the verifier a rule and asks for the sequence it produces\. The same prompt is used for the seed, the equivalent and the rivals, which is what makes the three comparable\.EQUIV\_PROMPTasks for other statements of the same rule,RIVAL\_PROMPTfor rules that reproduce a given prefix and then depart from it, andREPAIR\_PROMPTreturns an item to the composer with the fault named\.

The unquestionability and annotation modules include a series of prompts for costing mostly\.

The remaining prompts are for marking\.JUDGE\_PROMPTin the evaluator shows the prefix, the intended continuation and a reply, and asks how many terms were produced in order before the first mistake\. The annotator has a version of the same prompt that also sees the previous attempt, and answers whether more terms were derived this time, which is how a further token budget is granted\.TAU\_PROMPTreads a reply and reports where the answer begins, separating the tokens spent reasoning from the tokens spent answering\.

Finally, the examinee seesEXAM\_PROMPTand nothing else\. It is shown in full in[Figure14](https://arxiv.org/html/2610.14005#A9.F14)\.

Continue the sequence with the next \{k\} terms only\. Sequence: \{seq\} Think if you need to, but end your reply with the answer alone on the final line, the \{k\} terms separated by commas and nothing else\. Next \{k\} terms:

Figure 14:EXAM\_PROMPTis the prompt shown to examinees\.\{seq\}is the prefixx1:nx\_\{1:n\}, comma separated, and\{k\}iskfk\_\{f\}, which is 5 throughout\.The examinee is not told the rule, the domain, or how many terms the item holds\. The instruction about the final line allows a reply to be marked without parsing the model’s reasoning\.

## Appendix JCode and Main Files

The code is organised into seven python notebooks that follow the pipeline of[Figure1](https://arxiv.org/html/2610.14005#S5.F1)and[Section5](https://arxiv.org/html/2610.14005#S5)\. They are independent files that exchange information through files, containing items, programs, results, etc\.[Table33](https://arxiv.org/html/2610.14005#A10.T33)gives the inputs and outputs of each notebook\.

Three of the notebooks take one reference machine per run and write in the same file under a tag derived from its name, leaving the entries of other machines untouched\. This means that a fourth machine can be added later without repeating any earlier work\. Unquestionability generates independent files, but as it is writing onctest\_battery\.json, they are not parallelisable\.

The files are held as a dataset, refreshed at the start of each run and pushed at the end \(sometimes at checkpoints\), to be able to run on environments that are erased between sessions \(e\.g\., Colab or Kaggle code\)\.

Table 33:What each notebook reads and writes\.<tag\>is the reference machine:qwen3\-14b,qwen3\-8borphi\-4\-reasoning\. Every notebook appends one row toctest\_runs\.csv, which records the GPU/CPU times, wall clock times, the build, the settings and the prompt hashes\.
## Appendix KParameters

[Table34](https://arxiv.org/html/2610.14005#A11.T34)lists the parameters that are used to manoeuvre the execution of the code\. The generator values control how many items are attempted and how much is spent on each\. The annotator values control how much work a candidate description is allowed before it is recorded as having failed\. These parameters allow for an adaptive series of runs to populate the battery, certifying as many items as possible while keeping the use of the resources under control\.

Table 34:The main settings of the generator, the annotator and the evaluator\.settingvaluewhat it doesGeneratorN\_PER\_CELL1–3items wanted in each\(r,t\)\(r,t\)cellOVERSAMPLE12candidates composed per item wantedTERMS\_MIN/MAX28/72length of a sequence, by the load it carriesK\_ADMIN12terms held back, so an item can be administered at any prefix up to 60EXTEND\_REPS3cold runs of a rule, at temperatures 0\.0, 0\.3 and 0\.6EXTEND\_WINS2of those runs that must agree on every term for the item to be validREPAIR\_ROUNDS5times a faulty item is returned to the composer before it is droppedN\_EQUIVALENTS8other statements of the rule requestedN\_RIVALS6rivals requested at each prefixRIVAL\_PREFIXES5, 13prefixes rivals are always requested at, plus two from the estimate,L^\+2\\hat\{L\}\+2and1\.8​L^\+21\.8\\,\\hat\{L\}\+2, capped at 60THEME\_REUSE1items allowed to use one themeCALIBRATE\_CAP\_R0correction allowed on the retrieval target; 0 means the target is left as it isCALIBRATE\_CAP\_T2the same for transformationAnnotatorK\_MAX20terms the reference machine is asked to producebudgets1500, 3000, 6000token budgets for an item with a short prefix; 3000, 6000, 12000 for a long onehint\_budgets1500, 3000the same for a hint, which is tried before the full descriptionsTOP\_N3cheapest candidates run per itemRESCUEona description that nearly succeeded is retried with more roomBUDGET\_JUDGEgemini\-3\.5\-flash\-litedecides whether to grant a further budgetSCORING\_JUDGEgemini\-3\.6\-flashcounts the terms produced before the first mistakeEvaluatorkfk\_\{f\}5terms an examinee is asked forREPS3attempts per item per examineeexaminee budget2048, 8000, 16000token budget for an examinee’s reply: 16000 for GPT\-5\.6\-sol, Claude Opus 5, Sonnet 5 and Fable 5\.1, 8000 for the other API models, and 2048 for OLMo\-2\-7B and Qwen2\.5\-1\.5B
## Appendix LDisagreements between the Measure and the Scores

In[Table3](https://arxiv.org/html/2610.14005#S7.T3)we sawK​t\|Kt^\{\|\}reaches−0\.79\-0\.79Spearman correlation against accuracy over the 51 items certified under Qwen3\-14B\. The items where it disagrees with the scores are informative, because the kinds of disagreement have different causes\. Let us inspect them\.

### L\.1Items with highK​t\|Kt^\{\|\}and good scores

\.

K​t\|Kt^\{\|\}is a minimum over the candidate descriptions that were actually run\. The annotator tries a subset of conditional hints and unconditional equivalents from a finite set, and is also limited by budget constraints, so many candidates that would have cost less are sometimes never executed until the end\. The first cause of a highK​t\|Kt^\{\|\}and good scores can therefore originate from a search that stopped, and the value we report forK​t\|Kt^\{\|\}is a loose upper bound\. This case can be minimised with more search for candidates, and some exploration with higher budget managed to reduce the measuredK​tKtof0076,0134,0112and0004, which seem overestimated when looking at[Figure4](https://arxiv.org/html/2610.14005#S7.F4)\. Note that a possible shift to lower values would likely make the slopes of the characteristic curves steeper and the abilities slightly lower, which could even make the supra windows larger, not smaller\.

A second cause is a description the reference machine cannot derive from memory and must be named, i\.e\., a failed retrieval\. The cost is long under the reference machine, and its length measures the table, instead of the lower difficulty of actually retrieving it by other examinees that can derive it from memory\. Bothcℳ​\(p\)c\_\{\\mathcal\{M\}\}\(p\)andcℳ\(p∣x1:n\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:n\}\)come out high while the item is easy\. This happens at the top of our range, but less often than it may seem\. Of the 10 items above 190 bits, two, 0004 and 0112, have descriptions that spell out tables and are solved on most attempts by the six strongest examinees\. The other eight are answered correctly on at most 19% of attempts overall\. Six of them have five or more operations, and in the remaining two the description is long but the examinees fail as well\. Accuracy also falls through every band ofK​t^\|\\widehat\{Kt\}^\{\|\}, including the top one: the six strongest examinees average 98%, 67%, 65%, 37% and 19% in the bands 0–50, 50–100, 100–150, 150–190 and above 190 bits\. More details in[AppendixM](https://arxiv.org/html/2610.14005#A13)

The third cause is rare\. Two certified items haveτ\\tauequal to their budget, their runs reached the budget with the answer already in their working, soτ\\tauis slightly too high, by 0\.02 bits in the one we checked by hand\. A short description may need a lot of computation, but runs out of tokens in the reference machine, forcing the certification at a much more expensive, but efficient description \(e\.g\., one that enumerates some of the terms or intermediate steps\)\. In other words, we pay description bits for execution time\.

### L\.2Items with lowK​t\|Kt^\{\|\}and poor scores

This direction matters more, because it suggests the item is easy for the reference machine but difficult for the rest of the world\.

One possible cause is the tokeniser\. One item may have an advantageous tokenisation for the reference machine that does not work for other models\.

The serious case is a rule that names a one\-way function\. The first byte of a standard hash, or a stream cipher keyed on a word, can be stated in a line\. Consequently,K\|K^\{\|\}is small, while the terms will not reveal any information about the rule\. Also, a good cipher is inexpensive to evaluate forward, which is a design requirement, and the difficulty is in inversion, whichτ\\taudoes not measure\.

What keeps such items out of this battery is that the reference machine is a language model with no interpreter\. It cannot evaluate a hash forward either, so no description of that kind certifies\.

One way of keeping that situation at bay could be to say explicit that no cryptographic or pseudorandom number generators should be used\. But this would still allow for similar behaviours by unexpected \(almost\) one\-way functions\.

Interestingly, we calculate the conditional complexity given the prefix,K​t\|Kt^\{\|\}, and this can be compared with the unconditionalKKorK​tKt\. For a pseudorandom sequence the prefix is by construction indistinguishable from noise and the two costs coincide\. A description that is inexpensive and gains nothing from the prefix therefore has the profile of a one\-way item, and this could be a signal to inspect or exclude the item\.

Four of the 51 items certified for Qwen3\-14B have that profile, with a conditional cost below 80 bits and the two costs within 20 bits of each other\. Three of the four are easy, at 58%, 83% and 86% success\.

This comparison has been performed withcℳ​\(p\)c\_\{\\mathcal\{M\}\}\(p\)for appthat verifies with𝒱\\mathcal\{V\}andcℳ\(p\|x1:n\)c\_\{\\mathcal\{M\}\}\(p\|x\_\{1:n\}\)of appthat verifies withℳ\\mathcal\{M\}\. Future work should measure both things with the same verification machine, minimising both metrics separately and then calculate the difference\. In general, this is a traditional objection to Kolmogorov complexity as difficulty, as approximations ofKKorK​tKtare overestimations, but this case is an underestimation\. But removing those easy instances no model can solve would boil down to surrender to the populational approach we aimed to solve in this paper\. It may simply be that some stronger models start cracking the item with lowK​tKteventually, reconcilingK​tKtwith the observed difficulty\. Or there is a new concept that renders the apparently chaotic sequences into a nice pattern, as happened in science with many phenomena that looked chaotic, from the movement of the planets to hereditary traits\. Indeed, it is not known for certain that one\-way functions really exist: this is “arguably the most important problem in computer theory”\([Levin, 2003](https://arxiv.org/html/2610.14005#bib.bib55)\)\.

## Appendix MThe upper end of the Qwen3\-14B scale

Appendix[L\.1](https://arxiv.org/html/2610.14005#A12.SS1)notes thatK​t^\|\\widehat\{Kt\}^\{\|\}can be high because the reference machine cannot recall a table and has to be given it in the description\. For the Qwen3\-14B scale, the reference machine that has the widest supra\-window, there may be concerns that the items there are idiosyncratic\. Let us explore these items in more detail\.

At the top of the scale there is the first question of whetherK​t^\|\\widehat\{Kt\}^\{\|\}there measures the difficulty of the item or only what Qwen3\-14B does not know, especially for retrieval\. If it were mostly the latter, the items at the top would be easy for the examinees that have more general knowledge, partly because they have more parameters\. This is not what we observe from the data\.

Table 35:The 10 items above 190 bits under Qwen3\-14B, with the measuredr^\\hat\{r\}andt^\\hat\{t\}, the kind of description that certified them, and the share of attempts answered correctly by all twelve examinees and by the six strongest \(gpt5sol, opus\*, fable\*, sonnet\*, g3\.8F and g3\.6F\)\.Table[35](https://arxiv.org/html/2610.14005#A13.T35)lists the 10 items above 190 bits\. Only two of them,0004and0112, fit the concern\. Their descriptions spell out tables, the pig materials and the days of the week in one and the chess pieces and the rainbow colours in the other, and the six strongest examinees solve them on most attempts\. These are two of the items that[SectionL\.1](https://arxiv.org/html/2610.14005#A12.SS1)already identifies as overestimated, and for which further search lowered the measured value\. The other eight are answered correctly on at most 19% of attempts overall\. Six of them have five or more operations, so their cost comes from transformation and not from retrieval\. In the remaining two the description is long but the examinees fail as well\. In0146it spells out four short lists of words, and the examinees answer correctly on 1 of 36 attempts\.

Table 36:Average accuracy by band ofK​t^\|\\widehat\{Kt\}^\{\|\}under Qwen3\-14B, over the 51 certified items\.Table[36](https://arxiv.org/html/2610.14005#A13.T36)shows the same from the other side\. Accuracy falls through every band, and the top band is the hardest, for the strongest examinees as for all of them\. If the top of the scale mostly reflected what the reference machine does not know, the top band would not be the one the examinees fail most\.

Three further points bear on the question\. First,K​t^\|\\widehat\{Kt\}^\{\|\}is relative to its reference machine by design, as any measure of Kolmogorov complexity is relative to its universal machine\. What the machine does not know costs bits to state, and this is the crystallised component that the retrieval dimension keeps on purpose\. An examinee that knows a table saves those bits, as a person who knows the planets finds a sequence of planets easy\. Second, the choice of reference machine barely changes how the examinees are ranked\. Qwen3\-8B and Phi\-4\-reasoning, which differ from Qwen3\-14B in size and in lineage and training data respectively, order the twelve examinees almost as Qwen3\-14B does, at Spearman\+0\.99\+0\.99and\+0\.92\+0\.92\(Appendix[F](https://arxiv.org/html/2610.14005#A6)\)\. This would be unlikely if the scale reflected the particular gaps of one machine\. Third, as an estimate of the reference machine’s own trueK​tKt,K​t^\|\\widehat\{Kt\}^\{\|\}is an upper bound, and it can only err upwards\. A description the search did not find makes the value higher than it should be, never lower\. A longer search lowers such values, as further exploration with a larger budget did for0004and0112\.

Several reference machines can also be combined by taking, for each item, the smallestK​t^\|\\widehat\{Kt\}^\{\|\}among the machines that certify it, which removes the gaps of any one machine\. On the 62 items certified by at least one of the three machines, this minimum comes from Qwen3\-14B for 32 items, from Phi\-4\-reasoning for 24 and from Qwen3\-8B for 6\. Its Spearman correlation with accuracy is−0\.79\-0\.79, as strong as that of Qwen3\-14B alone on its 51 items, and the abilities it gives correlate with ECI at Pearson\+0\.81\+0\.81and Spearman\+0\.76\+0\.76\. At the top it changes nothing, since each of the 10 items above 190 bits is certified by Qwen3\-14B alone, so its value there rests on that machine\. Testing the top in the same way needs a larger battery with more reference machines\. This combination could be done in other ways, giving more weight to the reference machines in their areas of highest discrimination, or based on diversity to minimise the subjectivity or bias or choosing one single machine\. The infra\- and supra\-intelligence windows were introduced to illustrate that we may need a staircase of reference machines to cover wider windows\.

The most expensive certified items also have conditional costs close to their unconditional ones \([Table41](https://arxiv.org/html/2610.14005#A16.T41)\)\.0146costs 367 bits given the prefix and 364 without it,0084costs 290 and 303, and0139costs 243 and 263 \(all these numbers withoutτ\\tau\)\. The heuristic in Appendix[L\.2](https://arxiv.org/html/2610.14005#A12.SS2)flags items whose description is inexpensive and gains nothing from the prefix, because for them a lowK​t^\|\\widehat\{Kt\}^\{\|\}would hide a hard item\. Here the descriptions are expensive, soK​t^\|\\widehat\{Kt\}^\{\|\}is already high, and the examinees fail these items, as it predicts\. Across the 51 certified items, the less the prefix saves, the harder the item\. The shareK^\|/K^\\widehat\{K\}^\{\|\}/\\widehat\{K\}correlates with accuracy at Spearman−0\.61\-0\.61, and the 16 items for which the prefix saves less than a tenth of the description average 22% accuracy, against 86% for the 15 continued with no description at all\. A prefix that reveals little about the rule is what makes an inverse problem hard, andK​t^\|\\widehat\{Kt\}^\{\|\}records it\. Of the four items that do have the profile of Appendix[L\.2](https://arxiv.org/html/2610.14005#A12.SS2), with both costs below 80 bits and within 20 bits of each other, the only one the examinees fail is0133, whose short prefix admits a simpler reading, the nursery rhyme \([Table41](https://arxiv.org/html/2610.14005#A16.T41)\)\. The conditional cost can also exceed the unconditional one by a few bits, as in0146, because the two are computed with different prompts and checked by different machines, as Appendix[L\.2](https://arxiv.org/html/2610.14005#A12.SS2)notes\.

## Appendix NUnconditional Cost in the Unquestionability Filter

The previous discussion about one\-way functions and the idea of comparing the conditional and unconditional costs, as well as the problem of an item that was certified at 5 because the unconditional program worked, suggests some tension in our pipeline, combining unconditional complexity for unquestionability and conditional complexity for certification\. Why do not we just use the conditional version for all?

The difficulty is conditional\.Kℳℳ\(xn\+1:n\+k∣x1:n\)K\_\{\\mathcal\{M\}\}^\{\\,\\mathcal\{M\}\}\(x\_\{n\+1:n\+k\}\\mid x\_\{1:n\}\)usescℳ\(p∣x1:n\)c\_\{\\mathcal\{M\}\}\(p\\mid x\_\{1:n\}\), with the sequence first and the program after\. The conditioning is the examinee’s situation, and allows for partial hints that are sufficient when the prefix discloses much of the pattern already\. A sequence starting with the planets does not need to code the planets as a hint, as they are obvious\.

Unquestionability, on the other hand, usescℳ​\(p⋆\)\+δc\_\{\\mathcal\{M\}\}\(p^\{\\star\}\)\+\\delta, with both terms unconditional, as in[Corollary4\.1](https://arxiv.org/html/2610.14005#S4.Thmcorollary1)\. Conditioning would destroy the comparison, for the following reason\.

Among the programs that reproducex1:nx\_\{1:n\}and then diverge there is always the listing rival, which printsx1,…,xnx\_\{1\},\\dots,x\_\{n\}and then prints anything other thanxn\+1x\_\{n\+1\}\. Costed unconditionally it comes to aboutβ​n\+c0\\beta n\+c\_\{0\}, which grows withnn\. That growth is what makes the necessary condition[Definition4\.2](https://arxiv.org/html/2610.14005#S4.Thmdefinition2)\. a statement about prefix length, and what allows a longer prefix to exclude rivals\.

Costed conditionally onx1:nx\_\{1:n\}, the listing rival is nearly free, because a language model can copy tokens it can already see, with a ‘listing’ program such as “repeat the input forever”, which is consistent with the input by definition\. Thencℳ\(listing∣x1:n\)≈0c\_\{\\mathcal\{M\}\}\(\\text\{listing\}\\mid x\_\{1:n\}\)\\approx 0for everynn, the budgetcℳ​\(p⋆\)\+δc\_\{\\mathcal\{M\}\}\(p^\{\\star\}\)\+\\deltais never small enough to exclude it, and every item is questionable at every prefix whatever its rule\. The quantity stops depending on the item\.

The two frames answer different questions\.cℳ\(⋅∣x1:n\)c\_\{\\mathcal\{M\}\}\(\\cdot\\mid x\_\{1:n\}\)asks how hard the item is for a machine that has seen it, which is the difficulty\.cℳ​\(⋅\)c\_\{\\mathcal\{M\}\}\(\\cdot\)asks how short a program is in absolute terms, which is what a claim about all inexpensive programs has to quantify over\. An unquestionability claim concerns programs written by someone who was never shown the sequence\.

The implementation follows this\. The unquestionability notebook costs the rivals, the least expensive program and the listing rival withcℳ​\(p\)c\_\{\\mathcal\{M\}\}\(p\), unconditionally\. OnlyK\|K^\{\|\}andK​t\|Kt^\{\|\}, which are explicitly conditional, use the sequence as context\.

## Appendix OComputing Resources Used and Needed for Reproduction

We report the resources used by the project and an estimate of what a clean reproduction would need\. Both are approximate, and we give each as a central value with a range\.

The GPU time is taken from the weekly quotas of the notebook services we used, since each notebook’s own log records a run only when it ends normally and so misses every run that was stopped, timed out or failed\. Over the period of the logged runs the quotas give 355 GPU ‘T4’ hours on one service, and the other adds between 30 and 67 hours, depending on our estimates as this first service did not record usage per project\. Constrained by our goal of making the test sustainable and reproducible, all GPU work ran on a single NVIDIA T4\. Laptops in use for about ten hours a day per month in total, for orchestration and writing\. Part of the code and the analysis were developed assisted by AI assistants, OpenAI 5\.6Sol, Gemini 3\.1Pro, Claude Opus 5 \(mostly\) and 5\.5, over about 6,500 model calls\. We count its use from the saved transcripts: about 10 million tokens produced, between 7 and 13 million once the reasoning that is not saved is included, and about 1\.9 billion tokens of context read back across the calls, most of it from the providers’ caches when available, asking the models or estimating from our own logs and recollection\. Remote \(API\) model calls are counted in tokens, from the log for the composer, verifier, typer and judges and from the results file for the examinees\. Because some runs are missing from the log, including those for earlier versions of the battery, we take the recorded tokens as a lower bound, and twice that as an upper bound\.

Table 37:Resources used by the project\. Tokens are in millions and include reasoning tokens\. Energy and emissions are estimated under the assumptions in the text\.We follow the AI Energy Score initiative\([Luccioni et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib27)\), which reports the energy of inference per thousand queries and converts energy into environmental impacts with factors from the ADEME Base Empreinte database\. Energy differs on the source\. We assume 100 W while the GPU is attached, which is the T4’s 70 W limit and a share of the host, multiplied by 1\.1 for the data centre’s cooling and power supply\. We assume 45 W for laptops, with a range of 30 to 60 W\. The remote calls cannot be measured from outside, and the models we called are proprietary, so they are not among those AI Energy Score has measured\. For them we assume 0\.3 Wh per thousand tokens received, including the provider’s overheads, with a range of 0\.1 to 1\.0 Wh, and a tenth of that for tokens sent\. Over the 30,107 remote calls that were recorded, this comes to about 1\.2 kWh per thousand queries, a high figure because most of our queries asked for long reasoning\. For the assistant we use the same assumption for the tokens it produced, and a hundredth of it for context read from the cache, which is reused and not recomputed\. This adds about 9 kWh, between 2 and 33 kWh\. The remote calls account for most of the uncertainty, and at the central value for about half of the energy\.

For impacts we use the worldwide factors from AI Energy Score too, which are 9\.99 MJ of primary energy and7\.378×10−87\.378\\times 10^\{\-8\}kg Sb eq of abiotic resource depletion per kWh\. The project’s central estimate of 122 kWh then corresponds to 1\.2 GJ of primary energy and 9 mg Sb eq\. For emissions we use 402 g of CO2equivalent per kWh, the factor AI Energy Score gives for combined\-cycle natural gas turbines\.

Table[38](https://arxiv.org/html/2610.14005#A15.T38)estimates the resources for building and evaluating a battery of 100 valid items without failures or restarts\. The rates come from our measurement from runs that completed\. Generating 100 valid items takes about 140 candidates, since 71 per cent of the saved items were valid\. Each reference machine then costs every item, types it, and annotates the items it finds unquestionable, about 70 of the 100\. Annotation dominates the GPU time, at 25 to 50 minutes an item, most of it on items that no description certifies and so run through the whole ladder of token budgets\. The evaluation covers the items unquestionable under any machine, about 80, for twelve examinees reached through an API at three attempts each and two local examinees at one attempt\.

Table 38:Estimated resources for a clean reproduction on 100 valid items\. GPU hours are central values with ranges\. Tokens are in millions\. Other hours are CPU hours mostly\. The last two rows give the totals for one reference machine and for the three used in this paper\.With three reference machines, the estimate is about 130 GPU hours, 55 million tokens, between 15 and 68 kWh, between 0\.15 and 0\.68 GJ of primary energy, and between 6 and 27 kg of CO2equivalent\. The project used about three times as much GPU time and nearly four times as many tokens, the rest going to development, failures and restarts\. Two reasons account for much of the difference and can be turned into lessons learnt\. A reference machine that reasons at length needs a rule that stops its reply once the answer has appeared, or every reply runs to the token budget\. And a local examinee decoded greedily gives the same reply each time, so one attempt is enough\.

A reproduction needs one GPU with 16 GB of memory, which is enough for the reference machines in 4\-bit quantisation, somewhere to keep the state files between sessions, and API access to Gemini 3\.6 Flash and 3\.5 Flash\-Lite for the composer, verifier, typer and judges and to the provider of each examinee\. At the rate reported by the generator, the Gemini calls cost about 140 US dollars\. The cost of the examinees depends on which are chosen\. The twelve used here received about 6 million tokens between them\. On a single GPU the work takes about five and a half days without interruption, but generation and evaluation can be parallelised, and different reference machines can be costed separately\. A reproduction does not need the AI assistant, since the code is released\.

In summary, the numbers in this section support responsible and sustainable research, performing the experiments that are necessary for our research questions, and for research questions that matter\. Also, the whole point of the paper was to show that we can evaluate frontier models with a small number of items, and their generation has a reasonable cost, and energy consumption\.

## Appendix PThe Full Battery

We generated 151 items\. This had to be, by design, higher than the effective items in the battery\. While all the 151 items are kept, only the 107 valid items matter: the description of the remaining 44 were never consistently replicated\. Then, for each particular reference machine, only a subset became unquestionable and from them, the actual battery for eachℳ\\mathcal\{M\}, which only contains those that are also certifiable on that machine\. Many reasons other than unquestionability explain why items fall in the process, and the comparison of the full 151\-item list with the lists of certified items shows some biases\. Apart from the tokens required for reproducing the sequence during inference \(thinking\) time, the tokeniser may also explain why many sequences that were expressed with non\-alphanumerical patterns, such as ‘—\*—” struggled to make the cut\. More powerful reference machines could compensate for that, or in a future, tokenisers may change or even disappear, leading to a more natural notion of term as a finite string of characters rather than tokens\.

Let us show first a summary of domains and operations in[Table39](https://arxiv.org/html/2610.14005#A16.T39)\. We see that cuisine, farming, sport, law and folklore top the retrieval list\. Then, for operations, different kinds of sum and accumulations, modulus, multiplication, etc\., top the transformations list, while string transformation such as reverse appear in mid table\. Note that these are the mechanisms that can be named\. There are some other specific transformations that do not fall in any of the categories\.

Table 39:A summary of the 107 valid items in the battery \(out of 151 in total\) by domain of the theme \(as used by the retrieval\) and operation \(as used by transformation\)\. The left column counts items by the domain of the theme they were built from\. The column sums up to 107\. The right column counts items whose rule contains a given kind of operation, as found by ther,tr,t\-measurement procedure \(i\.e\., according tot^\\hat\{t\}and the shortest unconditional program\)\. An item may contain several, so the right column sums to more than 107\.The next, massive[Table40](https://arxiv.org/html/2610.14005#A16.T40)includes the 107 valid items, the id, their originalr0r^\{0\}andt0t^\{0\}, theme, first terms, and seed\. We see a diversity in structure, content and mechanisms\. Of course, some are small variations, such as item 0100 being a simple enumeration, or the theme being mere decoration, such as 0126\. But a sequence such as “o\-1, o\-2, …” may represent a distractor for very weak models, as compared to the trivial “1, 2, …”, so a number of items of this kind are beneficial\.

Table 40:The 107 valid items\.r0r^\{0\}andt0t^\{0\}are the retrieval and transformation steps the cell asked for, not the measured ones\. A value of−1\-1means the cell set no target for that component\. The theme was a prompt to the composer and does not always describe the result\. We show the first few terms as an examinee would see them, and the seed is the rule as the composer wrote it, both shortened to fit\.idr0​t0r^\{0\}t^\{0\}themefirst termsseed00020 1embroidery thread strand counts0 skeins 1 strands, 0 skeins 2 strands, 0 skeins 3 strands, 0 skeins 4 strands, 0 skeins 5 strands, 1 skeins…term n from 1 is n divided by 6 as ’a skeins b strands’00031 4bovine milk fat percentagesh pct fat, he pct fat, li pct fat, be pct fat, c pct fat, o pct fatA ledger starts with fat percentages \[1, 3\]\. Pop the lowest, emit its element symbol and ’ pct fat’\. Add double the percentage modulo 118 as a new debt, filter out…00042 5three little pigs building materialsrtsy11, itsy24, irby38, rtsy51, itsy63, irby76Start total at 0\. For step n, take the n\-th pig material \(straw, sticks, brick\) and n\-th day \(monday to sunday\)\. Add their combined lengths to total\. Each term is t…00051 2baseball field position numberspitcher, catcher, first base, second base, third base, shortstopcount from 1 skipping numbers with 0; replace each digit with its baseball field position00060 1islamic daily prayer sequence2, 6, 10, 13, 17, 19running total of raka’at for the daily prayers: 2, 4, 4, 3, 4 repeating00070 2airport runway magnetic headings02, 04, 06, 08, 10, 12from n=1, add n to its digit sum, take modulo 36 \(0 as 36\), formatted as 2 digits00092 2finger names on handbmuht alf, xedni bra, elddim cha, gnir del, yknip ech, bmuht foxPair fingers thumb to pinky with NATO alphabet; reverse the finger name and truncate the NATO word to 3 letters\.00101 2decathlon event competition schedule100 metres, long jump, shot put, high jump, 400 metres, 110 metres hurdlesqueue of the decathlon events in order\. emit front event; if day 1 \(first 5\) requeue once, else twice\.00111 0crop rotation plot yearshydrogen, helium, lithium, beryllium, boron, carbonthe chemical elements in order of atomic number00121 2american wire gauge size numbers4, 7, 9, 10, 11, 13american wire gauge sizes in order cycling from 4/0 to 40; each term is the running total of their digit sums00140 4circle of fifths keynotesg1, e1, c1, a1, f1, d1Queue of notes c,d,e,f,g,a,b and k=1\. Split queue into ceil\(len/2\) and rest, swap halves, pop front note: emit with k, shift letter forward by 4 notes, append to ba…00160 0espresso roast level darknessagtron 99, agtron 98, agtron 97, agtron 96, agtron 95, agtron 94count from 1; each term is agtron followed by a space and 100 minus n00202 1canonical hours of prayermatins h, lauds eh, prime il, terce eb, sext b, nones ccanonical hours of prayer in order, each followed by a space and the reversed element symbol by atomic number00210 5standard egg weight gradingsp, s, s p, m, m p, m sEgg grades p,s,m,l,x,j\. Start empty\. Each step: 1\. Push p to track A\. 2\. Merge adjacent equal wagons \(p\+p\-\>s, s\+s\-\>m, m\+m\-\>l, l\+l\-\>x, x\+x\-\>j\)\. 3\. Shunt heaviest to…00230 0natural counting numbers1, 2, 3, 4, 5, 6count from 100241 3stellar spectral classification lettersb, g, a, k, f, ofrom n=1; square n, reverse digits, take remainder mod 7, and take that 0\-based item from stellar spectral types00250 6us constitutional amendment numbersamendment 2, amendment 5, amendment 10, amendment 8, amendment 8, amendment 10count n from 1; square n, sum its digits, reverse the digits, take remainder modulo 27, add 1, prepend ’amendment ’00260 4candy cooking temperature stages0\-0\-1, 0\-0\-0, 0\-0\-0, 0\-1\-0, 1\-2\-0, 4\-2\-0Vessels Thread \(cap∞\\infty\), Ball \(5\), Crack \(3\) start at 0\. At step n: 1\) Add n to Thread; 2\) Fill Ball from Thread; 3\) Fill Crack from Ball, then empty Crack if full;…00271 1circle of fifths keysc 1, f 1, bb 1, eb 1, ab 1, db 1circle of fifths major keys from c using flats, repeated; each followed by a space and cycle number00290 6circle of fifths key signaturesc, g, d, a, e, bFrom n=0, fifths step 4n mod 7 gives letter from cdefgab, appended with floor\(n/7\) plus signs \+00300 3crochet hook letter designationsb hook, e hook, j hook, h hook, h hook, j hookcount n from 1; square n, sum its digits, and emit that letter of the alphabet starting at b with ’ hook’00310 3plimsoll line draft marks10m w, 12m tf, 16m w, 24m tf, 31m f, 36m tfdraft starts at 10; next is draft \+ digit sum \+ 1\. Term is draft, ’m ’, and mark from tf, f, t, s, w, wna by draft mod 600320 1crochet stitch height order12345, 12360, 12372, 12387, 12408, 12423a\(1\) = 12345; a\(n\) = a\(n\-1\) \+ the digit sum of a\(n\-1\)00340 5rainbow light spectrum colorsorange 1, orange 8, violet 27, red 28, red 35, indigo 54From n=1: square n, sum its digits, multiply by n; emit the spectrum color \(0=red, 1=orange, 2=yellow, 3=green, 4=blue, 5=indigo, 6=violet\) for product mod 7 and th…00410 6polygon side counts3, 4\-3, 5\-3\-4, 6\-4\-3, 7\-3\-4, 8\-4\-3Main track and siding start empty\. At step n \(n=1,2…\), wagon with polygon side count n\+2 arrives: 1\. Odd wagon goes to siding, even to main track\. 2\. Sort siding…00421 4tree ring annual counts1 mm, 4 mm, 8 mm, 10 mm, 12 mm, 15 mma tree trunk starts at 0 mm; each year add \(letter count of the nth chemical element mod 4 \+ 1\) mm; emit total with ’ mm’00432 4planets distance from sunmercury delta, venus echo, earth foxtrot, mercury golf, venus hotel, earth indiamercury, venus, earth start at track cells 1, 2, 3\. Each step, advance the lowest token by its planet rank to the next empty cell, emitting planet and cell’s NATO w…00451 0ten in the bed countdownduke, marquess, marquess, earl, earl, earlbritish peerage ranks in descending order with growing run lengths00461 1leather thickness in ounceshydrogen 1 oz, helium 2 oz, lithium 3 oz, beryllium 4 oz, boron 5 oz, carbon 6 ozthe chemical elements in order, each followed by a space and its atomic number in oz00470 4standard poker hand ranks1 one pair, 2 straight, 3 royal flush, 4 four of a kind, 5 four of a kind, 6 royal flushfrom n=1: square n, sum its digits, take remainder mod 10, add 1, and emit n with that poker hand rank00491 3months of the yearyra1, yra2, hcr3, lir4, yam5, enu6from n=1; take the nth month name in order, reverse it, keep the first 3 letters, and append n00502 0british peerage rank orderduke, red, marquess, orange, earl, yellowalternate between british peerage ranks in descending order and colors of the rainbow in order00541 5united kingdom peerage rank orderkx1, sxs2, l3, nt4, xrn5, x6For step n, cycle UK peerage ranks: reverse, replace vowels with x, slice first \(n%3\+1\) chars, rotate left 1, append n\.00582 3us presidential election yearsbbdf, qodf, jodf, ssbf, csbf, bsbfus presidential election years in order; substitute digits using ’republican’ \(0=r, 1=e, 2=p, 3=u, 4=b, 5=l, 6=i, 7=c, 8=a, 9=n\), reverse the string, and shift each…00611 6rebar diameter designation numbers1 00100, 2 01110, 3 11001, 4 00111, 5 11100, 6 10011Rule\-30 cellular automaton on a 5\-cell ring from 00100; each term is term number and binary state\.00641 6norse myth nine realms1:1\|, 2:\|21, 3:3\|21, 4:\|2143, 5:5\|2143, 6:21\|4365A,B=””,””\. s=str\(1\+\(n\-1\)%6\)\. Even len\(A\+B\): A\+=s; odd: B\+=s\+A\[:1\]; A=A\[1:\]\. While len\(B\)\>4\+n//10: A\+=B\[0\]; B=B\[1:\]\. A=A\[\-5:\]\. Output n:A\|B00662 1first verse of jabberwockybrillig7, aries5, slithy6, taurus6, toves5, gemini6alternate the nonsense words of Jabberwocky’s first verse with the zodiac signs from aries; each term is the word with its letter count appended00672 5the opening of hamletbernardo\>francisco:6, horatio\>marcellus:4, bernardo\>ghost:7, francisco\>horatio:5, bernardo\>marcellus:8, franc…5 Hamlet Act 1 Scene 1 characters in appearance order start in a queue with initial scores 5 to 1 from 5 bridge suits in descending order\. Each step, dequeue the fr…00721 0united states court hierarchyjay, rutledge, cushing, wilson, blair, iredellthe us supreme court justices by surname in order of appointment00731 2beer bitterness ibu ratings6 ibu, 12 ibu, 18 ibu, 24 ibu, 30 ibu, 36 ibuthe chemical elements in order; each term is 5 times the atomic number plus its digit sum, followed by ’ ibu’00742 1soil horizon layer letterso\-h, a\-he, e\-li, b\-be, c\-b, r\-ccycling master soil horizon letters \(o, a, e, b, c, r\) joined by hyphen to chemical element symbol by atomic number00762 5three billy goats gruff sizesred march, orange march, yellow june, green april, blue june, indigo april9\-cell ring ooooxoooo: step Rule 30 CA\. Term k emits k\-th rainbow color and lit\-count\-th month\.00771 3prologue to canterbury talesor1, wor2, r3, wr4, wyor5, or64 colors \(W=1<Y=2<O=3<R=4\) start left; boat moves 2 coolest right, 1 coolest left; reset when empty; left\+index00820 7sumo wrestling division ranks\*\*\#\#, \*\#\#\#, \*\*\#\#, \*\#\#\#, \*\#\#\#, \#\#\#\#for step n \(from 1\): 1\. select sumo division by \(n\-1\)%6 \+ 1; 2\. append n; 3\. reverse string; 4\. replace vowels with 1\-based alphabet positions; 5\. sum all numeric d…00830 7cardinal direction pointsw \-1 0, n \-1 1, w \-2 1, e \-1 1, e 0 1, w \-1 1Start at \(0,0\)\. For n=1,2…: 1\. square n, 2\. sum digits, 3\. multiply by 3, 4\. mod 4, 5\. map 0:n, 1:e, 2:s, 3:w, 6\. step 1 unit to \(x,y\), 7\. emit ’<dir\><x\><y\>’\.00841 7snooker ball point valuesred 1, yellow 9, green 21, brown 71, blue 81, pink 21For term n from 1, take the n\-th snooker ball in point\-value order \(1: red, 2: yellow, 3: green, 4: brown, 5: blue, 6: pink, 7: black, cycling every 7\)\. Multiply n…00861 7thirty two compass pointswwww\-2, tttttt\-4, wwwwww\-6, tttt\-8, –10, tttt\-12Cycle wine table\. Add vowels to running sum \(start 0\)\. Repeat 1st letter sum^2 mod 10 times \(\- if 0\), then \-sum\.00891 7coffee roast level gradations…\*…, \.\.\*…\., \.\.\*…\., …\.\*\.\., \*……, \.\*…\.\.for term n from 1: lookup roast level at n mod 8 \(1: green, 2: cinnamon, 3: new england, 4: american, 5: city, 6: full city, 7: vienna, 8: french\); remove vowels; c…00900 7primal beef cut locations\*chuck, obrisket, \.shank, \#rib, \*plate, \#loinfrom n=1, select the n\-th cut from \[chuck, brisket, shank, rib, plate, loin, flank, round\] cycling every 8; take its length; square it; add n; sum the digits; take…00911 7marathon kilometer milestone markerskm1\>, km2 v, km3\>, km4 ^, km5\>, km6\>For kilometer index n from 1, take the n\-th chemical element by atomic number; count the letters in its name; add n; square the result; sum the digits of the square…00950 0olympic medal rank order1 gold, 2 silver, 3 bronze, 4 gold, 5 silver, 6 bronzecount n from 1; each term is n, a space, and gold, silver, bronze in repeating medal rank order00960 0solomon grundy weekday eventsborn 1, christened 2, married 3, took ill 4, grew worse 5, died 6solomon grundy events in order repeating, each with day count n00970 0plywood veneer quality gradesa/a/a, a/a/b, a/a/c, a/a/d, a/b/a, a/b/b3\-ply veneer grades using letters a to d in alphabetical order, slash\-separated00981 0mohs hardness scale benchmark mineralstalc a, gypsum b, calcite c, fluorite d, apatite e, orthoclase fthe Mohs hardness scale benchmark minerals in order repeating, each paired with the alphabet in order00990 \-1the chemical elements by atomic number, as symbols, sta…na, mg, al, si, p, sthe chemical element symbols in lowercase by atomic number, starting from sodium0100\-1 \-1the years, counting up from some year in the last centu…1900, 1901, 1902, 1903, 1904, 1905the years in order, counting up from 190001010 \-1the letters of a very well known passage or title, one…t, o, b, e, o, rthe letters of ’to be or not to be that is the question’ in order01022 9international maritime signal flags\[dn\*8\], \[hc\*4\], \[r\*n9\], \[\*\*s5\], \[l\*k10\], \[l\*g6\]For term n starting at 1: multiply n by 5; add 3; sum the digits; if the digit sum is even select from 26 ICS flag names \(alfa to zulu, 0\-indexed\) at digit sum mod…01031 7wheels on the bus actionswednesday beep, wednesday beep, tuesday swish, friday move, sunday shh, tuesday swishs starts at 1\. For step n from 1: square s, add n, sum the digits, reverse those digits, take remainder mod 7 to get index i\. Retrieve the day at index i from the d…01042 8climbing route difficulty gradesf61, f63, f06, f08, h501, f09At step n \(1\-indexed\), retrieve the n\-th French grade \(cycling 15 items: 5a, 5b, 5c, 6a, 6a\+, 6b, 6b\+, 6c, 6c\+, 7a, 7a\+, 7b, 7b\+, 7c, 7c\+\) and n\-th YDS grade \(cycli…01050 7geological epoch chronological orderhol, mio, ple, hol, ple, mioFrom n=1: square n, add n, sum digits, multiply by 7, reverse digits, take modulo 7, map 0\.\.6 to pal, eoc, oli, mio, pli, ple, hol\.01063 9olympic podium place medals\#\#, \#\.\., \#\.\., \., \., \#\.At step n \(1\-indexed\), cycle through Olympic medals \(gold, silver, bronze\), motto \(citius, altius, fortius\), and ring colors \(blue, yellow, black, green, red\) to ge…01073 9hungry caterpillar fruit sequence\[\>\], \[<\], \[^\], \[^\], \[<\], \[<\]For step n from 1, take nth season \(spring\.\.winter\), month \(january\.\.december\), zodiac \(aries\.\.pisces\) mod table lengths\. Concatenate lowercase, drop vowels a,e,i,o…01082 9usda soil texture classesbeer\-14 ibu, beer\-14 ibu, beer\-16 ibu, beer\-13 ibu, beer\-12 ibu, beer\-19 ibuStep n: 0\-based floor\(\(n\-1\)/2\)%12 in T1\(Jan\-Dec\) if n odd else T2\(Aries\-Pisces\); drop aeiou; count chars; \*n; sum digits; %50; \+10; prepend ’beer\-’; append ’ ibu’\.0109\-1 8seasons of the year0\-1\-0\-0, 0\-1\-0\-3, 0\-1\-0\-54, 0\-2\-0\-54, 0\-2\-0\-105, 0\-2\-0\-108four seasonal counters \(spring, summer, autumn, winter\) start at 0; for step n from 1: square n, add n, sum digits, multiply by 5, reverse digits, mod 4 picks seaso…01102 2fibonacci sequence term values1 lph, 1 bt, 2 gmm, 3 dlt, 5 psln, 8 ztfor step n, join the n\-th fibonacci number mod 10, a space, and the n\-th greek letter with vowels dropped01113 1this old man rhyming targetsthumb\-do\-mercury, shoe\-re\-venus, knee\-mi\-earth, door\-fa\-mars, hive\-sol\-jupiter, sticks\-la\-saturnjoin with hyphens the cycling terms of: This Old Man targets \(thumb to home\), solfege \(do to ti\), and planets outward01123 2chess move ply counts15 plies red rook, 36 plies orange knight, 54 plies yellow bishop, 180 plies green queen, 0 plies blue king,…For term n from 1, take the nth back\-rank chess piece, its point value, and the nth rainbow color\. Multiply the value by the color length and n, emitting ’<product\>…01131 6rainbow color spectrum order1, 22, 63, 44, 51, 63For term n \(from 1\), take the n\-th color from the rainbow colors in spectrum order \(red, orange, yellow, green, blue, indigo, violet, cycling every 7\): 1\. count its…01141 3glass bead millimeter diameters3mm, 5mm, 7mm, 10mm, 13mm, 16mmfrom n=1, take the nth standard round glass bead diameter in millimeters cycling through 2, 3, 4, 6, 8, 10, 12, 14, 16, 18, 20; add n; take remainder modulo 19; app…01151 4electrical wire gauge numbers160000 ga, 360000 ga, 360000 ga, 160000 ga, 25 ga, 144 gafor step n from 1, take the nth gauge from standard AWG wire gauges in slash notation \(4/0, 3/0, 2/0, 1/0, 1, 2…\); replace ’/’ with ’0’ to get an integer; multipl…01162 7usda beef quality gradespc\-04, cr\-25, sb\-61, sr\-96, cf\-01, us\-00S starts at 0\. For step n=1,2…: 1\. multiply n\-th USDA quality grade position \(1\-8\) by \(3n\-2\)\-th beef primal cut position \(1\-8\); 2\. add n; 3\. sum digits; 4\. add to…01171 6nautical knot speed incrementso\-o\-o\-o, none, o\-o, o\-o\-o\-o\-o, o\-o, o\-o\-o\-o\-o\-oFor step n from 1, take the n\-th zodiac sign \(cycling 12\)\. Count its consonants, add n, multiply by 3, floor divide by 2, take mod 7, and print that many ’o’s joine…01181 2baking flour extraction percentages46%, 56%, 66%, 81%, 111%, 151%the standard French flour types in ascending order, repeating; each term adds the 1\-based cycle count to the type number and appends ’%’01193 4us felony class levels\#, \#\#, \#\#\#, \#\#\#\#\#\#\#\#, \#, \#\#\#Alternate in order through three tables \(cycling each\): US federal felony classes, US federal misdemeanor classes, and US federal prison security levels\. At term n…01203 4meat doneness internal temperaturesa s r\*d 5, b w \*r\*ng\* 1, c n y\*ll\*w 7, d e gr\*\*n 5, e s bl\*\* 4, a w \*nd\*g\* 8Cycle A\.\.E \(ranks 5\.\.1\), N,E,S,W \(opposites S,W,N,E\), 7 rainbow colors\. Stock=10\. Step: stock=\(stock\+9\-rank\)%9\. Emit class, opposite dir, color \(vowels \*\), stock\.01213 4bovine permanent teeth emergence1\#\#, 1\#\#\#, 1\#\#\#\#, 2\#\#\#\#\#\#\#\#, 2\#\#\#\#\#\#\#\#\#, 2Concatenate tables T1, T2, T3 \(len 30\)\. For n=1\.\.72: v=L\[\(\(ceil\(n/3\)\-1\) mod 30\)\+1\]\. Ignore non\-digits in v\. Append \(\(digit\_sum\(v\)^2\+n\) mod 10\) ’\#’ chars to v\.01221 2sheep shearing pattern steps1y, 1h, 1l, 1k, 1d, 1ksheep shearing steps \(belly, crutch, tail, neck, head, flank, side\); term is pass number plus reversed step truncated to pass01233 0michelin star tier levels1 star, sweet, monday, 2 stars, sour, tuesdayalternating between michelin star tiers, basic taste modalities, and days of the week in order01241 \-1the whole numbers, counting up in steps of the same siz…delaware, pennsylvania, new jersey, georgia, connecticut, massachusettsthe us states in order of admission to the union01253 0tractor power take off speeds540 rpm, type 1, mercury, 750 rpm, type 2, venusalternate: pto speeds \(540 rpm, 750 rpm, 1000 rpm\), iso pto categories \(type 1, type 2, type 3, type 4\), planets outward0126\-1 0us military officer rankso\-1, o\-2, o\-3, o\-4, o\-5, o\-6the letter o followed by a dash and n from 101272 0numbers from one to tenone, triangle, two, quadrilateral, three, pentagonalternate between English numbers from one to ten and polygons with 3 to 10 sides in order01281 \-1the whole numbers, counting up from any startgenesis, exodus, leviticus, numbers, deuteronomy, joshuathe books of the protestant bible in order01293 1opening stanzas of the ravenwea, mer, jan, lor, ven, febalternate in order through the line\-ending words of stanza 1 of The Raven, the planets outward, and the months of the year, truncating each word to its first three…01301 \-1the letters of a very well known passage or title, one…c, a, l, l, m, ethe letters of the opening of Moby Dick in order, one per term01313 1archery target score ringswe, pn, as, bk, kt, tsalternate through archery ring colors by score, chess pieces by value, and zodiac signs in order; emit first and last letters01323 \-1the chemical elements by atomic number, as names, start…sodium, mercury, january, magnesium, venus, februaryAlternate in order: element from Sodium, 8 planets Mercury\-Neptune, month January\-December\.0133\-1 \-1the words of a very well known song or nursery rhyme, o…1 mary, 2 had, 3 a, 4 little, 5 lamb, 6 marystep number and repeating words of ’mary had a little lamb’01343 0tailor seam allowance widths1/8 in, mercury, pacific, 1/4 in, venus, atlanticCycle 3 lists: 7 seam allowances \(1/8 in\.\.1 in\), 8 planets \(Mercury\.\.Neptune\), 5 oceans \(Pacific\.\.Arctic\)\.01354 2decathlon event order1, s, a, m, ll, aaCycle through decathlon events, summer Olympic host cities since 2000, Greek alphabet letters, and planets in order; each term takes the first character of the item…01363 1snow white dwarf namesbas, red, mer, doc, ora, venAlternate in order through the seven dwarfs alphabetically, rainbow colors in order, and solar system planets outward from the Sun; truncate each word to three lett…0137\-1 7espresso shot extraction times120s, 287s, 22s, 117s, 284s, 19sv starts at 25; for term n from 1: add n to v, multiply by 7, sum digits, square, subtract n, take absolute value, set v to the result, emit v with suffix ’s’01381 7beekeeping frame super depths\[======\], \[======\], \[=======\], \[=======\], \[======\], \[=======\]For step n \(1\-indexed\), take the n\-th planet from the sun \(cycling\)\. Count its letters, square the count, add n, sum the digits, take mod 4, add 5, and render that…01390 7goldilocks porridge temperatures69c: just right, 64c: just right, 64c: just right, 89c: too hot, 89c: too hot, 64c: just rightt starts at 70; at step n \(from 1\): 1\) multiply t by n, 2\) sum digits, 3\) square, 4\) take modulo 100, 5\) add 20 to set new t, 6\) classify t \(<55: too cold,\>85: too…01403 2us state admission orderjhd, fhp, mln, abg, mbc, jcmfor term n from 1 to 50, take the nth US state in admission order, the nth chemical element by atomic number, and the nth month of the year \(cycling\); extract the i…01410 7hokey pokey body partshead, left foot, right hip, head, head, right hipcount n from 1; square n; add 7; sum digits; multiply by 5; reverse digits; take remainder mod 8 to index \(0: right foot, 1: left foot, 2: right hand, 3: left hand,…01421 7the four seasons ordergn\*rps1, r\*tn\*w2, gn\*rps3, mn\*t\*\*4, mn\*t\*\*5, gn\*rps6At step n \(1\-indexed\): square n, sum its digits, add 3, take mod 4 to pick the season \(0=spring, 1=summer, 2=autumn, 3=winter\), reverse the season name, replace vow…01434 2the gettysburg addressuof, saw, led, naj, ocs, adaCycle in turn through four tables: gettysburg address words in order \(no punctuation\), us presidents by surname, us states by admission order \(no spaces\), and month…01443 2jack and jill rhymejader, anegnaro, jiwolley, weneerg, upeulb, thogidniCycle 25 words of Jack and Jill stanza 1, 7 solfege notes, and 7 rainbow colors\. Join rhyme word \(truncated to note length\) with reversed color\.01452 7twelve labors of hercules\#\#\#\#\#\#\#\#\#, \#, \#, \#\#\#\#\#\#\#\#\#, \#\#\#\#\#\#\#\#\#, \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#Step n: append lowercase 3\-letter month \(\(n\-1\) mod 12 \+ 1\) to lowercase element symbol n; reverse; pick 1\-based even chars; sum letter values \(a=1\); square; sum dig…01464 2opening of a christmas carolnamygrelc1, yelram2, egoorcs3, tnelis4, krelc5, tsap6Cycle 4 tables \(clergyman,clerk,undertaker,mourner; marley,past,present,future; scrooge,marley,fred,cratchit; silent,jingle,deck,joy\) round\-robin; reverse word, app…01472 \-1the letters of a very well known passage or title, one…\.\.\-\., —, \.\.\-, \.\-\., …, \-\.\-\.for each letter of the gettysburg address in order, its international morse code01481 \-1the whole numbers, counting up in steps of the same siz…i, ii, iii, iv, v, vithe Roman numerals in ascending order from 101490 0united states pledge of allegiancei, pledge, allegiance, to, the, flagthe words of the us pledge of allegiance in order, lowercase without punctuation01502 0musical tempo marking namesgrave, do, largo, re, adagio, mialternate between the 8 main tempo markings in order of speed and solfege syllables in scale order01510 0the poem the ravendo1, re1, mi1, fa1, sol1, la1solfege syllables do to ti in order with an incrementing octave numberFinally,[Table41](https://arxiv.org/html/2610.14005#A16.T41)refers to the previous table through id, and for the three reference machines it shows V, U or C representing if the item has reached valid, unquestionable or certified status, and other columns with several complexity metrics, the estimatedrrandtt, the split point for unquestionability \(LUL\_\{U\}\), etc\. We see that the shortest description found become cryptic sometimes, often expressed in pseudocode\. This is on purpose, as this is the conditional version, and explains why “British peerage and ROYGBIV colors” works\. It is not sufficiently unambiguous to generate the whole sequence from scratch but a sufficient hint for the continuation\. This makes the difference between unconditional and conditional programs more clear\.

We checked all the items externally as a postprocess, once all the results were processed, to avoid cherry\-picking or introducing a manual step in the construction of the battery\. This post\-mortem check serves as validation, as we found very few errors\. Namely, just a few items admit a simpler reading of their prefix\. They also show up during evaluation, since the examinees then agree on the same wrong continuation, and the ranking is not altered\. Under Qwen3\-14B there are three of the 51 certified items \(0133,0110and0127\), under Qwen3\-8B three of 32, and under Phi\-4\-reasoning four of 38\. In0133the examinees continue the rhyme after “Mary had a little lamb”, and in0110the Fibonacci numbers are taken modulo 10 but the five terms shown are all below 10\. All of them passed at a prefix of 5 or 13 terms\. In addition, one item certified only under Phi\-4\-reasoning, 0142, repeats an arithmetic slip of the generator in 15 of its terms\. We keep all of them in the battery\. Removing them raises every correlation betweenK​t^\|\\widehat\{Kt\}^\{\|\}and accuracy, to−0\.84\-0\.84under Qwen3\-14B\. Such items could be caught before evaluation by taking the reference machine’s own unaided continuation as a rival, and by recomputing formal rules with a program\. It is important to highlight that these items lie between 40 and 146 bits ofK​tKtfor Qwen3\-14B, below the upper end of the scale, where the supra window is located\. Also, the previous check could be automated without relying on the examinees, so increasing the robustness of the battery would only require some extra compute\.

Table 41:Each of the 107 valid items of the battery\. The stage is the furthest each reached for each reference machine: V valid, U unquestionable, C certified\.KKisK^ℳ𝒱​\(x\)\\widehat\{K\}^\{\\mathcal\{V\}\}\_\{\\mathcal\{M\}\}\(x\), the cost of the least expensive description of the whole sequence;LuL\_\{u\}the prefix at which no rival survives;K\|K^\{\|\}the cost of the description that certified, zero where the machine continued the sequence unaided\.r^\\hat\{r\}andt^\\hat\{t\}are the measured steps under Qwen3\-14B\. The last column is the description that certified under Qwen3\-14B, shortened\.stageKKLuL\_\{u\}id14B8BPhi14B8BPhi14B8BPhir^\\hat\{r\}t^\\hat\{t\}K\|K^\{\|\}, and the description0002CCC10810410413550\.02\.00 \(none needed\)0003CUU2722712346060601\.05\.0187 L = \[1, 3\]; loop: x = min\(L\); L\.remove\(x\); L\.append\(2\*x % 118\); output lowercase period…0004CUU2342522616060602\.05\.0225 t=0; m=\[’straw’,’sticks’,’brick’\]; d=\[’monday’,’tuesday’,’wednesday’,’thursday’,’friday’,…0005CCC1141221091313421\.01\.00 \(none needed\)0006CCC7074711313131\.01\.00 \(none needed\)0007VVV116116107———0\.02\.2—0009CUU1431351205552\.02\.2141 nth term: reverse of \(thumb, index, middle, ring, pinky\)\[\(n\-1\)%5\] \+ ’ ’ \+ 3\-letter prefix of …0010VUU157167147—35351\.02\.2—0011CCC2828335551\.00\.00 \(none needed\)0012VUU141142140—27271\.03\.0—0014VUV113126129—18—0\.02\.2—0016CCC7887845551\.01\.00 \(none needed\)0020UUU1331411285552\.01\.0—0021VVV245232214———0\.03\.2—0023CCC2217205550\.00\.00 \(none needed\)0024VUC158156147—27271\.04\.0—0025VVU161169146——230\.04\.4—0026VVV166167164———0\.01\.0—0027VVV148179161———1\.02\.2—0029VVV131138135———0\.02\.4—0030CCU1541601462727270\.04\.0115 n ? \(1 \+ sum of digits of n2\)th letter of the alphabet \+ ’ hook’ \(n = 1, 2, …\)0031CCU2302252173636360\.03\.0169 d1=10; dn\+1 = dn \+ sum\_digits\(dn\) \+ 1; termn = dn \+ ”m ” \+ \(tf, f, t, s, w, wna\)\[dn mod 6\]0032CCC4156505550\.02\.00 \(none needed\)0034CCC1932041821313131\.04\.4128 n=1; repeat: p = n \* sum\(digits of n^2\); output ROYGBIV\[p % 7\] \+ ” ” \+ p; n \+= 10041VVV134126127———0\.01\.8—0042UVU12614512923—231\.03\.8—0043CCC126135131135351\.03\.00 \(none needed\)0045UUC9810394131351\.01\.0—0046CCC9910293135131\.81\.20 \(none needed\)0047VVV176174171———1\.04\.0—0049CCC1281151115551\.03\.039 Month names reversed0050CCC8792905552\.00\.054 British peerage and ROYGBIV colors0054CUU2032011801313131\.05\.0189 term\(n\) = rotate\_left\(1, vowel\_x\(reverse\(UK\_peerage\_rank\(n\)\)\)\[:n%3\+1\]\) \+ n0058CUU1801691621313131\.02\.0168 US presidential election years in order\|reverse digits\|0123456789 \-\>sfqvcmjdbo0061VVV151138140———0\.01\.8—0064VVV377406385———0\.06\.2—0066UUC163173163513132\.01\.8—0067VVV277290316———0\.03\.4—0072UUU6268645551\.00\.0—0073VVC127122110——130\.03\.6—0074CCC1551571451313131\.02\.847 soil horizon layer letters0076CUU2032102095050502\.02\.2177 k \-\>\(k\-th rainbow color\) \+ ” ” \+ \(lowercase month number M\[\(k\-1\) mod 8\]\), where M = …0077VVC147145138——130\.02\.0—0082VVV284298287———1\.06\.0—0083CUU2162082146060600\.05\.2201 x, y = 0, 0 for n = 1, 2, … d = ”nesw”\[3 \* sum\_digits\(n^2\) % 4\] x \+= \(d==’e’\) \- \(d==’w’\) …0084CUU3033112886060601\.05\.2290 For term n from 1, take the n\-th snooker ball in point\-value order \(1: red, 2: yellow, 3: green…0086VUC178184169—13130\.03\.4—0089UUU1631681803838380\.03\.8—0090UUU2972893106060600\.07\.0—0091UUC2012132115050501\.05\.2—0095VVV10498120———0\.01\.8—0096CCC1251271321313131\.01\.80 \(none needed\)0097CCC75888851350\.00\.80 \(none needed\)0098CUC105114106131351\.01\.091 nth term: \(nth Mohs scale benchmark mineral\) \+ ” ” \+ \(nth letter of the alphabet\)0099CCC8478625551\.01\.00 \(none needed\)0100CCC3037305550\.00\.00 \(none needed\)0101CCC44504651350\.01\.037 Hamlet soliloquy0102UUU192191169131350\.04\.2—0103UUU2152281941313130\.06\.6—0104UUU3353543246060601\.05\.8—0105VVV215211217———0\.03\.0—0106VVV201198194———0\.03\.2—0107VVV311322302———3\.07\.6—0108CUU3203042985959592\.02\.0290 1\. Set k = floor\(\(n\-1\)/2\) mod 12\. 2\. Pick k\-th month of the year if n is odd, else k\-th zod…0109CUU2522452354040400\.05\.0180 c = \[0, 0, 0, 0\]; for n = 1, 2, …: r = rev\(5 \* digit\_sum\(n^2 \+ n\)\); c\[r mod 4\] \+= r; output c\[0\]\-…0110CUU1171191045551\.22\.8118 f\(n\) = \(F\_n mod 10\) \+ ’ ’ \+ \(n\-th Greek letter minus vowels\)0111CCC2322312301313133\.02\.092 Cycles: 10 number rhymes, 7 solfege, 8 planets0112CUU2362312261313133\.02\.2230 Each term n \(starting at 1\) is n×\\timeslen\(c\)×\\timesv ”plies” c p, where c is the nth rainbow color, p is…0113CUU1441381431313131\.06\.0127 n \-\>reverse of \(n \* digit\_sum\(\(length of n\-th rainbow color\)^2 \+ n\)\)0114VVV196188195———1\.03\.6—0115CCC1441601451313130\.03\.4119 \(\(n<5 ? 100\*\(5\-n\) : n\-4\) \* n\)^2 \+ ” ga” for n = 1, 2, …0116VVV402412406———2\.08\.4—0117VVV256264250———1\.03\.4—0118VVV168170154———1\.03\.4—0119VVV317334276———3\.03\.0—0120UUU2882842711313131\.04\.0—0121VCV171166170—22—0\.03\.2—0122VVV233249225———0\.04\.0—0123CCU1321281401313473\.01\.032 5 basic tastes0124CCC3447515551\.00\.036 US states in order of admission to the Union, in lowercase0125VVV173190179———1\.01\.0—0126CCC4245435550\.01\.055 US officer pay grades0127CUC90941001313391\.00\.084 Alternate endlessly: English numbers 1 to 10 and polygon names for 3 to 10 sides\.0128CCC4542525551\.00\.00 \(none needed\)0129CUU1861941861313133\.01\.0147 Interleave first 3 letters of: line\-ending words of stanza 1 of The Raven, planets outward from…0130UCC5146475551\.01\.0—0131UUU1481411321313133\.01\.0—0132CUC1341261331313133\.01\.089 Cycle through lowercase lists: chemical elements from sodium, planets from mercury, months…0133CCC8183805551\.01\.077 monotone counter paired with cyclic word list0134CUU1521451461313132\.01\.0149 Round\-robin emission from 3 lists: \[1/8 in, 1/4 in, 3/8 in, 1/2 in, 5/8 in, 3/4 in, 1 in\], 8 plan…0135VVV276281247———4\.02\.0—0136CUU1521631571313133\.01\.2135 Cycle in order through 7 dwarfs alphabetically, ROYGBIV rainbow colors, and planets from…0137VVV217238243———0\.07\.0—0138VVU211216198——431\.05\.4—0139CUU2632652575050500\.05\.8243 t:=70\. At step n=1,2,…: t := 20 \+ \(digit\_sum\(n\*t\)^2 % 100\)\. Emit ‘$\{t\}c: $\{t<55 ? ’too cold’…0140UUU1501511571313133\.02\.8—0141CUU2252362105555550\.05\.2160 n ? \{1, 2, 3, …\} ? \(right foot, left foot, right hand, left hand, head, right hip, left hip, whole…0142VVC238241208——130\.07\.6—0143UUU2052302081313134\.03\.2—0144UUU2482542481313133\.04\.2—0145UUU3072992926060602\.08\.0—0146CUU3643913581313130\.03\.0367 f\(n\) = reverse of W\[\(n\-1\)%4\]\[\(\(n\-1\)/4\)%4\] \+ n for n=1,2,… with W=\[\[clergyman,clerk…0147UCC6458615552\.00\.8—0148CCC5149515551\.00\.00 \(none needed\)0149CCU64687151351\.02\.038 natural text order0150UUU113116122135132\.00\.0—0151VVV104109120———0\.02\.2—
## Appendix QProof of[Corollary4\.1](https://arxiv.org/html/2610.14005#S4.Thmcorollary1)

Despite being trivial, we include the proof here for completeness:

###### Proof\.

Letβ​L\\beta Lbe the cost of the programpλp^\{\\lambda\}“printx1:Lx\_\{1:L\}, thenxL\+1:L\+kx\_\{L\+1:L\+k\}”, wherec0c\_\{0\}is the fixed overhead of the print template and is independent of the intended term\. Letz′≠xL\+1:L\+kz^\{\\prime\}\\neq x\_\{L\+1:L\+k\}be an alternative continuation\. This gives a rival programqqof the same costβ​L\+c0\\beta L\+c\_\{0\}\. Ifβ​L\+c0<c⁡\(p⋆\)\+δ\\beta L\+c\_\{0\}<c\(p^\{\\star\}\)\+\\delta, this program is a rival inside the window that continues differently, contradictingδ\\delta\-unquestionability\. Henceβ​L\+c0≥c⁡\(p⋆\)\+δ\\beta L\+c\_\{0\}\\geq c\(p^\{\\star\}\)\+\\delta, i\.e\.c⁡\(p⋆\)≤β​L−δ\+c0c\(p^\{\\star\}\)\\leq\\beta L\-\\delta\+c\_\{0\}\. ∎

## Appendix RAcknowledging Models, Data and Libraries Used

We used the following models, data and libraries, and we acknowledge their use\.

The reference machines were Qwen3\-14B and Qwen3\-8B\([Qwen Team, 2025](https://arxiv.org/html/2610.14005#bib.bib1)\)and Phi\-4\-reasoning\([Abdin et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib3)\), all loaded in 4\-bit NF4 quantisation\([Dettmers et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib20)\)\. Gemini 3\.6 Flash\([Google DeepMind, 2026c](https://arxiv.org/html/2610.14005#bib.bib15)\)was the composer, the verifier, the typer and the scoring judge, and Gemini 3\.5 Flash\-Lite\([Google DeepMind, 2026b](https://arxiv.org/html/2610.14005#bib.bib14)\)the budget judge\. The examinees were GPT\-5\.6 Sol\([OpenAI, 2026](https://arxiv.org/html/2610.14005#bib.bib8)\), GPT\-5 nano\([OpenAI, 2025](https://arxiv.org/html/2610.14005#bib.bib7)\), Claude Opus 5\([Anthropic, 2026c](https://arxiv.org/html/2610.14005#bib.bib11)\), Claude Sonnet 5\([Anthropic, 2026a](https://arxiv.org/html/2610.14005#bib.bib10)\), Claude Fable 5\.1\([Anthropic, 2026b](https://arxiv.org/html/2610.14005#bib.bib12)\), Claude Haiku 4\.5\([Anthropic, 2025](https://arxiv.org/html/2610.14005#bib.bib9)\), Gemini 3\.8 Flash, 3\.6 Flash, 3\.5 Flash\-Lite and 3\.1 Flash\-Lite\([Google DeepMind, 2026d](https://arxiv.org/html/2610.14005#bib.bib16);[Google DeepMind, 2026c](https://arxiv.org/html/2610.14005#bib.bib15);[Google DeepMind, 2026b](https://arxiv.org/html/2610.14005#bib.bib14);[Google DeepMind, 2026a](https://arxiv.org/html/2610.14005#bib.bib13)\), OLMo\-2\-7B\([Team OLMo et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib4)\)and Qwen2\.5\-1\.5B\([Qwen Team, 2024](https://arxiv.org/html/2610.14005#bib.bib2)\)\. Gemma\-3\-4B\([Gemma Team, 2025](https://arxiv.org/html/2610.14005#bib.bib6)\), Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib5)\)and OLMo\-2\-7B were also calibrated, as a check on the reference machines’ constants\.

Abilities were compared with Epoch AI’s Capabilities Index\([Epoch AI, 2025](https://arxiv.org/html/2610.14005#bib.bib17)\), whose data are released under CC BY 4\.0\. The local models ran with PyTorch\([Paszke et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib19)\), Transformers\([Wolf et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib18)\), Accelerate\([Gugger et al\., 2022](https://arxiv.org/html/2610.14005#bib.bib21)\)and bitsandbytes, with weights from the Hugging Face Hub, and the API models were called through the Google Gen AI, OpenAI and Anthropic Python libraries\. The analyses used NumPy\([Harris et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib22)\), SciPy\([Virtanen et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib25)\)and Matplotlib\([Hunter, 2007](https://arxiv.org/html/2610.14005#bib.bib23)\), in Jupyter notebooks\([Kluyver et al\., 2016](https://arxiv.org/html/2610.14005#bib.bib24)\)run on Kaggle Code and Google Colab\.

## Appendix SExtended Related Work

#### Inverse problems, inversion problems and one\-way functions\.

Inverse problems have always been quintessential to \(artificial\) intelligence\.[McCarthy \(2006\)](https://arxiv.org/html/2610.14005#bib.bib43)said that “getting reality from appearance is an inverse problem\. Geologists, oil companies, and astronomers are faced with inverse problems\. Their solution is intellectually difficult and computationally intensive\. Human\-level AI systems will also have to be able to infer reality from appearances related to them in complex ways”\. The term ‘inverse problems’ was traditionally understood in the context of education and scientific discovery\([Groetsch, 1999](https://arxiv.org/html/2610.14005#bib.bib52)\)\. In machine learning, it is now commonly associated with the restoration or recovering of a part of an image or other modality\([Zirvi et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib33)\)but its original conception as general inversion problems in science is still preserved with some benchmarks for LLMs\([Dou et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib34);[Zheng et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib31)\)\. The term ‘inversion problem’ usually has a more restricted perspective in computer science and mathematics, as the specific goal of inverting input\-outputs of a given function or machine\([Levin, 1973](https://arxiv.org/html/2610.14005#bib.bib65);[Levin, 2003](https://arxiv.org/html/2610.14005#bib.bib55);[Levin, 2013](https://arxiv.org/html/2610.14005#bib.bib54)\)\. Some of these problems, the so\-called one\-way functions, are problems that are very easy to compute in one direction, but extremely difficult and time\-consuming to reverse\. Its existence, not proven yet in its most precise formulation\([Levin, 2003](https://arxiv.org/html/2610.14005#bib.bib55)\), would imply that NP≠\\neqP, which would be reassuring to the notion of hard\-to\-solve easy\-to\-verify in which future AI evaluation may depend\.

#### Solutional complexity\.

Inverse problems are different for functions, sequences and worlds\. One goal is to find a short description of the world, and another to find a policy for that world that allows us to success or explain part of it\. In the C\-test the*instance generator*and the*acceptable policy*\(the solution\) for the ‘problem’ coincide; in interactive settings this should be different\.[Hernández\-Orallo \(2015\)](https://arxiv.org/html/2610.14005#bib.bib57)shows that aggregating tasks weighted by such a*difficulty function on policies*, rather than by a universal distribution over tasks, yields decompositions of which the C\-test is the natural non\-interactive special case\. This solutional view is developed at length by[Hernández\-Orallo \(2017\)](https://arxiv.org/html/2610.14005#bib.bib58), where task difficulty is characterised by properties \(size, running time and space, etc\.\) of the set of acceptable solutions \(given some tolerance\)\. Even if we are using sequences, the present paper adopts this view, made clear by the use of the conditional version of Levin’sK​tKt\. If we were to generate environments, we should look for the shortest description of an acceptable policy \(solution\) for the environment, not the shortest description of the environment, as in the Universal Intelligence metric from[Legg and Hutter \(2007\)](https://arxiv.org/html/2610.14005#bib.bib71), or its approximation\([Legg and Veness, 2013](https://arxiv.org/html/2610.14005#bib.bib72)\), where simple uninteresting environments dominate the measure, or make the concept of optimality weak\([Leike and Hutter, 2015](https://arxiv.org/html/2610.14005#bib.bib73)\)\.

#### Compression tests, LLMs as compressors and distilling Occam’s razor\.

Many ‘intelligence as compression’ tests exist\([Mahoney, 1999](https://arxiv.org/html/2610.14005#bib.bib40);[Dowe and Hájek, 1997](https://arxiv.org/html/2610.14005#bib.bib76);[Hutter, 2006](https://arxiv.org/html/2610.14005#bib.bib39)\), asking the examinee to compress the data\. The idea of an LLM compressing a distribution comes back to the very definition of language model given by[Shannon \(1949\)](https://arxiv.org/html/2610.14005#bib.bib106);[Shannon \(1951\)](https://arxiv.org/html/2610.14005#bib.bib107)\.[Jaimovitch\-López et al\. \(2021\)](https://arxiv.org/html/2610.14005#bib.bib69)asked whether LLMs identify simple algorithmic concepts from small witness sets\. In other words, they found that Occam’s razor can be*distilled*from data into the model’s prior\. They measured how GPT architectures, program\-induction systems, and humans perform as a joint function of concept complexity and number of examples\. In general, it is well recognised today that LLMs are good compressors\([Deletang et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib32)\), but the analysis of their power for small sequences for testing and the calibration of difficulty is only fully achieved in C\-test ONE\.

#### Item response theory, characteristic curves and Epoch’s ECI\.

Our derivation of abilities from a logistic fit follows the tradition of Item Response Theory \(IRT\)\([Embretson and Reise, 2013](https://arxiv.org/html/2610.14005#bib.bib88);[Bock and Gibbons, 2021](https://arxiv.org/html/2610.14005#bib.bib93)\), where the notion of difficulty plays a central stage\([Rust et al\., 2021](https://arxiv.org/html/2610.14005#bib.bib85)\)\. IRT contrasts difficulty and ability from the start, and sets a common scale for them\. Given a matrix of results of subjects and items, IRT infers the ability of the subjects and the difficulty of items at the same time\. Many traditional IRT models represent the probability of success as a logistic function of the difference between ability and difficulty\. The simplest of these models, the 1\-PL or Rasch model, only estimates ability and difficulty, assuming the logistic curve has slope 1\. The 2\-PL model extends this by allowing a second parameter per item, the discrimination, represented by the slope of the logistic curve\. These logistic curves are called item characteristic curves, with ability on thexx\-axis and response on theyy\-axis for a population of subjects\. The dual curve is called the person characteristic curve\([Vale and Weiss, 1975](https://arxiv.org/html/2610.14005#bib.bib101);[Trabin and Weiss, 1983](https://arxiv.org/html/2610.14005#bib.bib102)\), in this paper referred to as \(agent\) characteristic curve, which maps response on theyy\-axis for a population of items as a function of difficulty \(in our caseK​tKt\) on thexx\-axis\. An important thing to clarify is that IRT, unlike this paper, is populational on the subjects, and cannot predict performance for items that have not been characterised during the estimation phase, as the parameters depend on the population of subjects\. If the subjects or the items are changed \(very different groups of humans\), all the parameters may change\. IRT has been adapted to machine learning and NLP\([Martínez\-Plumed et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib87);[Lalor et al\., 2016](https://arxiv.org/html/2610.14005#bib.bib108);[Kipnis et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib90);[Maia Polo et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib89);[Vania et al\., 2021](https://arxiv.org/html/2610.14005#bib.bib91);[Lalor et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib94)\), and more recently with the famous ECI index\([Ho et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib92)\), a standard IRT application of a large and well\-curated matrix of AI models and benchmarks\. This IRT application is performed at the benchmark level, rather than the item level\. In our paper, difficulty is not inferred from a population but estimated from the items directly\. That is why we can anticipate difficulty even before no human or systems has yet solved the problem\.

#### Procedural generation and evaluating with video games\.

Procedural generation\([Cobbe et al\., 2020](https://arxiv.org/html/2610.14005#bib.bib28);[Jin et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib30);[Mirzadeh et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib95);[Voudouris et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib29)\)is common in assessment, and it is becoming more frequent in AI because of contamination\. Our use ofr,tr,tpairs is a kind of parametrised procedural generation, but the actual difficulty metric is then assigned by the program search\. Sequences have many properties that make it a right choice for procedural generation, but more complex mini\-worlds, where the solution is a policy, are likely to be the next step of this programme of evaluating AI using algorithmic information theory\. As mentioned in[Section2](https://arxiv.org/html/2610.14005#S2), there is a revival of evaluating AI with video games or other small worlds\([Mnih et al\., 2015](https://arxiv.org/html/2610.14005#bib.bib35);[Hu et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib38);[Paglieri et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib37);[Ying et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib46)\), but the major links are with those that are procedurally\-generated\([Pérez\-Liébana et al\., 2019](https://arxiv.org/html/2610.14005#bib.bib36);[Voudouris et al\., 2025](https://arxiv.org/html/2610.14005#bib.bib29);[Jin et al\., 2023](https://arxiv.org/html/2610.14005#bib.bib30);[Ying et al\., 2026](https://arxiv.org/html/2610.14005#bib.bib47)\)\.

#### Measurement theory and AI scales

Measurement theory\([Hand, 2016](https://arxiv.org/html/2610.14005#bib.bib83)\)deals, among other things, with the type ofscaleand the measurementunits\. Regarding the scales, Steven’s typology of measurement\([Stevens, 1946](https://arxiv.org/html/2610.14005#bib.bib100)\)includes ‘nominal’, ‘ordinal’, ‘interval’ and ‘ratio’ scales\. For quantitative measurement, interval scales make the comparisons of differences meaningful, and calculating a mean is well justified\. Ratio scales set an absolute value, and hence ratios become meaningful\. For a latent factor, the choice of a scale is arbitrary\. In IRT, it is somewhat controversial \(e\.g\.,\([Michell, 1999](https://arxiv.org/html/2610.14005#bib.bib96);[Lord, 1975](https://arxiv.org/html/2610.14005#bib.bib97)\)\)\. Although some researchers advocate for a ratio or interval\-scale basis for test scores \(e\.g\., ‘the odds \(or log odds\) of student 1 answering a class of items correctly are twice the odds \(or log odds\) of student 2 doing so’ is a meaningful assertion\([Freund, 2019](https://arxiv.org/html/2610.14005#bib.bib98)\)\), others point out that a latent trait scale of a fitted IRT model is convenient but not inherently meaningful because any monotonic transformation of the estimated latent trait scores produces an equally valid model\([Lord, 1975](https://arxiv.org/html/2610.14005#bib.bib97);[Wallmark et al\., 2024](https://arxiv.org/html/2610.14005#bib.bib99)\)\. In contrast to this, our magnitude is just bits, which is a ratio scale, with an absolute zero and fully additive\. Beyond algorithmic information theory\([Hernández\-Orallo, 2017](https://arxiv.org/html/2610.14005#bib.bib58)\), it is unusual to use bits as a unit for intelligence evaluation\. A major exception is the concept of perplexity \(or cross\-entropy\) in LLMs, inherited from[Shannon \(1948\)](https://arxiv.org/html/2610.14005#bib.bib86)\. While perplexity was a good proxy of capabilities in the early days of LLMs, the extensive use of post\-training stages and more recently reasoning makes it unfit for general capability evaluation\. And this will be more so in agential situation\. This paper reconnects evaluation, large language models and information coding in a different, yet similarly fundamental way\.