NxN E-valuation: Hypothesis Certification via a Conformal CRT Null

# Hypothesis Certification via a Conformal CRT Null
Source: [https://arxiv.org/html/2608.06621](https://arxiv.org/html/2608.06621)
## N×\\timesN E\-valuation: Hypothesis Certification via a Conformal CRT Null

Bin Wang Ads CoreML / LLM4Rec Meta binwang88@meta\.com&Yan Zhong Ads CoreML / LLM4Rec Meta yzhong36@meta\.com

###### Abstract

We proposeN×\\timesN E\-valuation, a handy, e\-value\-based hypothesis\-certification algorithm that lets a hypothesis be verified without building any case\-specific certification procedure—such as constructing a dedicated null hypothesis—as long as a large enough dataset is available\. The method is especially suited to LLM\-based exploration systems, where LLMs are remarkably good at*proposing*hypotheses but suffer badly from hallucination; this hallucination prevents us from harvesting LLM outputs directly, and existing remedies each fall short\. The most common solutions include letting the LLM verify or correct itself \(circular verification\(Huanget al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib5); Kamoiet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib6)\)\) and held\-out testing \(which false hypotheses can still pass via spurious correlations\(Yeet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib7)\)\), among other remedies detailed in the introduction\. To resolve this, N×\\timesN E\-valuation exploits the naturally existing large training set and lets different samples serve as null hypotheses for one another\. This design directly realizes a conditional randomization test \(CRT\)\(Candèset al\.,[2018](https://arxiv.org/html/2608.06621#bib.bib4)\)that certifies each hypothesis\. The approach can be a universally better replacement for at least LLM circular verification and held\-out\-data testing, provided the LLM’s generations are hypotheses that apply to each individual sample\.

## 1Introduction

Automated systems—LLMs above all—now*propose*hypotheses about data: an LLM abduces*why*a user chose an item, a program synthesizer emits candidate predictors, a miner proposes subgroup rules\. Proposal is no longer the bottleneck;certificationis\. LLMs are remarkably good at proposing yet hallucinate badly, which requires each proposal to be checked, and that check is frequently*as hard as the original problem*, or demands bespoke, case\-specific statistical machinery \(a dedicated null hypothesis, a hand\-built test\) for every new proposal\. We study this in general form: a stream of*per\-unit hypotheses*, each mapping a unit’s inputxix\_\{i\}to a per\-unit*restriction*scoring its held\-out outcomeyiy\_\{i\}, to be certified against real data\.

Existing remedies for this hallucination–verification problem each fall short; the growing use of LLMs and synthesizers to*generate*hypotheses\(Alkanet al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib13)\)has only sharpened the need\. We group them by*what establishes trust*\.

Ask a model to judge the output\.The most common remedy has the LLM*verify or correct itself*, and a close relative trains a separate*verifier*or process\-reward model to score answers or reasoning steps\(Lightmanet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib20); Zhanget al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib21)\)\. Both are bounded by the same weakness—the judge is a model of the same kind, inheriting the errors it is meant to catch and offering no external, data\-grounded guarantee; self\-correction often fails to improve, or even degrades, accuracy\(Huanget al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib5); Kamoiet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib6)\), and a learned verifier is itself fallible and gameable\.

Ground the output in an external oracle\.*Formal deduction*translates a claim into a proof assistant and machine\-checks it\(Google DeepMind,[2025](https://arxiv.org/html/2608.06621#bib.bib8); Songet al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib9)\);*domain\-model or knowledge\-graph grounding*ties reasoning to a trusted domain model, ontology, or curated database\(Fallahpouret al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib14); Amayuelaset al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib15)\);*retrieval with attribution*requires each claim to be supported by and cited to a corpus, with automatic fact\-checking\(Kanget al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib16); Zhaoet al\.,[2026](https://arxiv.org/html/2608.06621#bib.bib17)\); and*tool/execution grounding*verifies by running code, querying data, or acting in an environment\(Yaoet al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib18); Gaoet al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib19)\)\. Each is reliable within its niche but presupposes a domain\-specific oracle: it certifies formalizable claims, or consistency with already\-known knowledge, or the result of an executable check—not the empirical truth of an arbitrary, novel per\-unit hypothesis\.

Prevent bad outputs by construction\.Constraining or instructing the reasoning so only valid, grounded outputs can form—grammar\-constrained decoding\(Genget al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib11)\), or curated, format\-restricted reasoning traces distilled from a teacher\(OneRec Team,[2026](https://arxiv.org/html/2608.06621#bib.bib10)\)—raises reliability, but narrows what may be hypothesized and certifies*form*, not*truth*: a well\-formed, in\-scope output can still be false\.

Verify statistically on data\.Classical conditional\-independence tests—permutation tests and the model\-X CRT\(Candèset al\.,[2018](https://arxiv.org/html/2608.06621#bib.bib4)\), including sequential e\-value variants\(Shaeret al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib22)\)—certify relevance rigorously, but each needs a hand\-built or model\-X null*per hypothesis*; and when the hypothesis is itself*suggested by the data*, validity further demands characterizing the selection event\(Markovicet al\.,[2017](https://arxiv.org/html/2608.06621#bib.bib23)\)or spending data on a split—all intractable for an open, undocumented proposer\. The most common data\-based check, plain*held\-out validation*\(cross\-validation; the correctness test in discovery benchmarks\(Majumderet al\.,[2025](https://arxiv.org/html/2608.06621#bib.bib12)\)\), is only half of certification: it shows a hypothesis*predicts*but cannot separate a real per\-unit mechanism from a base rate, a prevalence effect, or a “Clever Hans” shortcut\(Yeet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib7); Geirhoset al\.,[2020](https://arxiv.org/html/2608.06621#bib.bib24)\), so false hypotheses pass\.*Human expert audit*is the reliable fallback but does not scale and is often as hard as the original problem\.

What is missing across all of these is a*general, automatic, distribution\-free*certifier for arbitrary per\-unit hypotheses—needing no case\-specific null, and separating “predicts” from “predicts for the right reason”: a genuine per\-unit mechanism, or a real scope\-level effect, as opposed to a base rate or a shared confound\.

Three properties make this hard and rule out standard tooling:

1. 1\.Open, un\-enumerable hypothesis space\.The hypotheses are being*discovered*; a method needing a fixed feature schema or a hand\-built null per hypothesis does not scale\.
2. 2\.Multi\-round, data\-dependent testing\.Hypotheses are proposed, refined, and re\-tested over rounds; naïve p\-value thresholding or a batch Benjamini–Hochberg pass either inflates false discoveries or shifts the threshold retroactively as the candidate set grows\.
3. 3\.Two distinct failure modes\.A hypothesis can \(a\) fail to beat a base rate \(not*significant*\), or \(b\) predict well for the*wrong reason*—a base rate, a prevalence effect, or a shared confound rather than a genuine*mechanism*\. Certification must rule out both\.

N×\\timesN E\-valuation solves these challenges with a large enough dataset, and it turns that dataset into the null: rather than hand\-building a null per hypothesis, we let different samples serve as null hypotheses for one another\. We form anN×NN\\times Ncross\-prediction matrixwhose entry\(i,j\)\(i,j\)scores unitii’s restriction on unitjj’s held\-out outcome\. Its diagonal is each hypothesis applied to its own unit; its off\-diagonal is the same restriction applied to*other*units—a per\-unit conditional randomization test \(CRT\)\(Candèset al\.,[2018](https://arxiv.org/html/2608.06621#bib.bib4)\)whose null isconstructed automatically from the dataand read off as an exact e\-value\. Because e\-values compose, the certificate staysanytime\-valid\(Ramdaset al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib1); Grünwaldet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib2)\)as data and discovery rounds accumulate\. The result is a drop\-in, broadly applicable replacement for at least LLM self\-verification and held\-out\-set testing, whenever the LLM’s generations are hypotheses that apply to each individual sample\.

Empirically \([Section5](https://arxiv.org/html/2608.06621#S5)\), on a synthetic world with planted ground truth the certifier assigns the correct verdict99%99\\%of the time, whereas naive held\-out predictive validation scores50%50\\%—it certifies every predictive rule and so is fooled by all four planted fakes\. The headline case is a*structural confound*that fools held\-out testing*and*a significance\-only ablation \(100%100\\%certify\) yet is rejected by the mechanism e\-value \(100%100\\%\)\. The theoretical guarantees—e\-value validity, the intersection–union error bound, anytime\-validity, and the identifiability of glocal effects from structural confounds—are developed in[Section4](https://arxiv.org/html/2608.06621#S4)\.

The remainder of the paper is organized as follows\.[Section2](https://arxiv.org/html/2608.06621#S2)formalizes the certification problem and the running instantiation\.[Section3](https://arxiv.org/html/2608.06621#S3)develops the method—the two nulls, the significance and conformal\-mechanism e\-values read off theN×NN\\times Ncross\-prediction matrix, the intersection–union decision, the treatment of unit\-invariant hypotheses, and multi\-bank accumulation\.[Section4](https://arxiv.org/html/2608.06621#S4)establishes the guarantees—e\-value validity, the certification\-error bound, anytime\-validity, and the identifiability of glocal effects from structural confounds\.[Section5](https://arxiv.org/html/2608.06621#S5)reports the synthetic\-validation experiments, and[Section6](https://arxiv.org/html/2608.06621#S6)concludes; additional experiment details are in[AppendixA](https://arxiv.org/html/2608.06621#A1)\.

## 2Problem Formulation

Table 1:Objects in the certification problem, with the running recommendation example\.The core concepts used in the formulation are listed in[Table1](https://arxiv.org/html/2608.06621#S2.T1)\. The setup fits most scenarios naturally: a hypothesis poses a condition \(the scope\) and predicts a characteristic for the subjects that satisfy it \(the restriction\)\. To certify a hypothesisHH, two conditions must hold of its restrictions on held\-out outcomes\.

*Significance\.*The restrictions must predict the outcome beyond the population base rate—the lift is real, not a base\-rate artifact\.

*Mechanism\.*The lift must be*unit\-specific*—it depends on*this*unit’s input, rather than a unit\-invariant effect that would hold for any in\-scope unit\.

We certifyHHonly when both hold, controlling false certifications across an open, multi\-round stream, and without needing to name*which*sub\-structure of the input drives the lift\.[Section3](https://arxiv.org/html/2608.06621#S3)operationalizes both criteria \([Algorithm1](https://arxiv.org/html/2608.06621#alg1)\);[Section4](https://arxiv.org/html/2608.06621#S4)establishes the guarantees\.

## 3Method: TheN×NN\\times NE\-Value Matrix Certifier

The algorithm scores each criterion with an*e\-value*\(Ramdaset al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib1); Grünwaldet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib2)\)and certifiesHHwhen both exceed the level\-α\\alphathreshold1/α1/\\alpha\. Beyond the restriction scoresri​\(⋅\)r\_\{i\}\(\\cdot\)\([Section2](https://arxiv.org/html/2608.06621#S2)\), it needs only the restriction’s base rate over the test setTTas a null\.[Algorithm1](https://arxiv.org/html/2608.06621#alg1)gives the procedure; the subsections below defineEsigE^\{\\mathrm\{sig\}\}andEmechE^\{\\mathrm\{mech\}\}and establish the guarantees\.

Algorithm 1N×NN\\times NE\-valuation: certify hypothesisHHat levelα\\alpha\(single evaluation;[Algorithm2](https://arxiv.org/html/2608.06621#alg2)adds bank extension\)1:

H=\(in​\_​scope,apply\)H=\(\\mathrm\{in\\\_scope\},\\mathrm\{apply\}\), data

\{\(xi,yi\)\}i∈T\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i\\in T\}, level

α\\alpha, diversity threshold

τdiv\\tau\_\{\\mathrm\{div\}\}
2:

U←\{i∈T:in​\_​scope​\(xi\)=1\}U\\leftarrow\\\{\\,i\\in T:\\mathrm\{in\\\_scope\}\(x\_\{i\}\)=1\\,\\\}⊳\\trianglerightin\-scope subsetU⊆TU\\subseteq T

3:for

i∈Ui\\in Udo

4:

ri←apply​\(xi\)r\_\{i\}\\leftarrow\\mathrm\{apply\}\(x\_\{i\}\)⊳\\trianglerightrestriction; high computation, once per unit \(O​\(\|U\|\)O\(\|U\|\)\)

5:endfor

6:for

i∈U,j∈Ui\\in U,\\ j\\in Udo

7:

M​\[i\]​\[j\]←ri​\(yj\)M\[i\]\[j\]\\leftarrow r\_\{i\}\(y\_\{j\}\)⊳\\trianglerightcheap score; diagonalM​\[i\]​\[i\]=ri​\(yi\)M\[i\]\[i\]=r\_\{i\}\(y\_\{i\}\)\(O​\(\|U\|2\)O\(\|U\|^\{2\}\)\)

8:endfor

9:

Esig←1\|U\|​∑i∈Uri​\(yi\)/g0\(ri\)E^\{\\mathrm\{sig\}\}\\leftarrow\\tfrac\{1\}\{\|U\|\}\\sum\_\{i\\in U\}r\_\{i\}\(y\_\{i\}\)\\big/g\_\{0\}^\{\(r\_\{i\}\)\}⊳\\trianglerightsignificance: diagonal vs\. base rate overTT

10:

Emech←1\|U\|​∑i∈Uri​\(yi\)/\(1\|U\|​∑j∈Uri​\(yj\)\)E^\{\\mathrm\{mech\}\}\\leftarrow\\tfrac\{1\}\{\|U\|\}\\sum\_\{i\\in U\}r\_\{i\}\(y\_\{i\}\)\\big/\\big\(\\tfrac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}r\_\{i\}\(y\_\{j\}\)\\big\)⊳\\trianglerightmechanism: diagonal vs\. in\-scope row mean

11:

div←1\|U\|​∑j∈Uσj/μj\\mathrm\{div\}\\leftarrow\\tfrac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}\\sigma\_\{j\}/\\mu\_\{j\}⊳\\trianglerightrestriction diversity \([Section3\.4](https://arxiv.org/html/2608.06621#S3.SS4)\)

12:return

verdict​\(Esig,Emech,div\)\\mathrm\{verdict\}\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\},\\mathrm\{div\}\)per[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)⊳\\trianglerightunit\-specific / glocal / global\-prevalence / reject

### 3\.1Two nulls

Significance nullH0sigH\_\{0\}^\{\\mathrm\{sig\}\}: under the restriction, an in\-scope outcome is exchangeable with the populationTT—the restriction is satisfied no more by in\-scope outcomes than across the test set as a whole\.

Mechanism nullH0mechH\_\{0\}^\{\\mathrm\{mech\}\}: conditional on the pools of restrictions and outcomes, the within\-scope pairing of restrictions to outcomes is exchangeable—the lift carries no*unit\-specific*information\.

The mechanism null probes structure at the*per\-unit*granularity—it randomizes pairings*within*UU\. A unit\-invariant hypothesis—one whose restriction is constant over its scopeUU\(glocal or global\-prevalence,[Section3\.6](https://arxiv.org/html/2608.06621#S3.SS6)\)—has no such structure: the within\-scope randomization is vacuous andEmech=1E^\{\\mathrm\{mech\}\}=1\. Yet it is not mechanism\-free; its “cause” is scope membership itself, so the only mechanism\-relevant contrast is whether in\-scope outcomes exceed the population base rate—exactly what the significance randomization \(UUagainstTT\) measures\. The mechanism question then rises one level, from the unit to the scope: the mechanism e\-value degrades to the significance e\-value, andEsigE^\{\\mathrm\{sig\}\}serves both roles—certifying both that the lift is real and that it is attributable to the hypothesis’s scope\. A unit\-invariant hypothesis is therefore judged onEsigE^\{\\mathrm\{sig\}\}alone—certified as*glocal*, or flagged*global\-prevalence*when it merely sits at the base rate \([Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)\)\.

### 3\.2Significance e\-value

Herer​\(y\)r\(y\)\([Section2](https://arxiv.org/html/2608.06621#S2)\) scores how well outcomeyysatisfies the restriction\. The significance baselineg0\(rj\)g\_\{0\}^\{\(r\_\{j\}\)\}is the restriction’s own rate over the whole test setTT—how often it is satisfied across the population\. For unitjjwithrj=apply​\(xj\)r\_\{j\}=\\mathrm\{apply\}\(x\_\{j\}\), the significance e\-value divides its score on the true outcome by that baseline,

Ejsig=rj​\(yj\)g0\(rj\),g0\(rj\)=1\|T\|​∑k∈Trj​\(yk\)\.E\_\{j\}^\{\\mathrm\{sig\}\}\\;=\\;\\frac\{r\_\{j\}\(y\_\{j\}\)\}\{g\_\{0\}^\{\(r\_\{j\}\)\}\},\\qquad g\_\{0\}^\{\(r\_\{j\}\)\}\\;=\\;\\frac\{1\}\{\|T\|\}\\sum\_\{k\\in T\}r\_\{j\}\(y\_\{k\}\)\.\(1\)Sincej∈Tj\\in T, the pool already containsyjy\_\{j\}, which makes the e\-value exact \(as inEmechE^\{\\mathrm\{mech\}\}\)\. This is a conditional randomization test whose null pool is the population: doesrjr\_\{j\}scorejj’s own outcome above a random unit’s? A hypothesis with no lift over the base rate getsEsig≈1E^\{\\mathrm\{sig\}\}\\approx 1; only signal beyond it accumulates evidence\.

### 3\.3Mechanism e\-value: theN×NN\\times Nmatrix and its conformal normalization

Over the claimed population \(size\|U\|\|U\|\), compute each restriction*once*, then form

M​\[i\]​\[j\]=ri​\(yj\)\(uniti’s restriction scored on unitj’s held\-out outcome\)\.M\[i\]\[j\]\\;=\\;r\_\{i\}\(y\_\{j\}\)\\qquad\(\\text\{unit $i$'s restriction scored on unit $j$'s held\-out outcome\}\)\.\(2\)The diagonal is the correct pairing; off\-diagonal entries apply a restriction to another unit’s outcome\. The per\-unit*conformal mechanism e\-value*\(Vovk and Wang,[2021](https://arxiv.org/html/2608.06621#bib.bib3)\)row\-normalizes the diagonal by its row \(the exchangeable pool\):

Ejmech=M​\[j\]​\[j\]1\|U\|​∑k∈UM​\[j\]​\[k\]\.E\_\{j\}^\{\\mathrm\{mech\}\}\\;=\\;\\frac\{M\[j\]\[j\]\}\{\\tfrac\{1\}\{\|U\|\}\\sum\_\{k\\in U\}M\[j\]\[k\]\}\.\(3\)Intuition: does unitjj’s restriction scorejj’s*own*outcome above a random unit’s? If not, the hypothesis is not unit\-specific\.

### 3\.4Diversity

A third statistic, read off the same matrix, records whether different units receive*different*restrictions\. Let

μj=1\|U\|​∑i∈UM​\[i\]​\[j\],σj=1\|U\|​∑i∈U\(M​\[i\]​\[j\]−μj\)2\\mu\_\{j\}\\;=\\;\\frac\{1\}\{\|U\|\}\\sum\_\{i\\in U\}M\[i\]\[j\],\\qquad\\sigma\_\{j\}\\;=\\;\\sqrt\{\\frac\{1\}\{\|U\|\}\\sum\_\{i\\in U\}\\big\(M\[i\]\[j\]\-\\mu\_\{j\}\\big\)^\{2\}\}\(4\)be the mean and standard deviation of columnjj\(the mean and spread of a restriction’s scores across items\); diversity averages each column’s coefficient of variationσj/μj\\sigma\_\{j\}/\\mu\_\{j\},

diversity=1\|U\|​∑j∈Uσjμj\.\\mathrm\{diversity\}\\;=\\;\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}\\frac\{\\sigma\_\{j\}\}\{\\mu\_\{j\}\}\.\(5\)UnlikeEsigE^\{\\mathrm\{sig\}\}andEmechE^\{\\mathrm\{mech\}\}, diversity is not an e\-value: it reads off whether the restrictions differ*as scored on the observed outcomes*, or are constant, and thereby routes the decision \([Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)\)—a constant restriction is judged onEsigE^\{\\mathrm\{sig\}\}alone, while a diverse one must also clear the mechanism CRT\. In particular it separates a unit\-invariant real hypothesis from a structural confound, whichEsig,EmechE^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\}alone cannot \([Theorem2](https://arxiv.org/html/2608.06621#Thmtheorem2)\)\. Its thresholdτdiv\\tau\_\{\\mathrm\{div\}\}is calibrated empirically and carries no Type\-I guarantee, but the certificates it routes to do\. Being a tolerance rather than an exact zero, it admits as constant those restrictions that differ only slightly, or on few outcomes;[AppendixB](https://arxiv.org/html/2608.06621#A2)analyses whatDDresolves and what it does not\.

### 3\.5Certification decision

Aggregate the per\-unit e\-values by their mean, itself an e\-value under arbitrary dependence \([Section4](https://arxiv.org/html/2608.06621#S4)\):

Esig=1\|U\|​∑j∈UEjsig,Emech=1\|U\|​∑j∈UEjmech\.E^\{\\mathrm\{sig\}\}=\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}E\_\{j\}^\{\\mathrm\{sig\}\},\\qquad E^\{\\mathrm\{mech\}\}=\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}E\_\{j\}^\{\\mathrm\{mech\}\}\.\(6\)The signals combine into a verdict withEmechE^\{\\mathrm\{mech\}\}as the primary axis—Emech≥1/αE^\{\\mathrm\{mech\}\}\\geq 1/\\alphamarks a genuine per\-unit mechanism \(*unit\-specific*\), whileEmech≈1E^\{\\mathrm\{mech\}\}\\approx 1marks its absence \(the*unit\-invariant*family\), among which diversity confirms a constant restriction andEsigE^\{\\mathrm\{sig\}\}fixes the sub\-type:

verdict​\(H\)=\{unit\-specificEmech≥1/α∧Esig≥1/α,glocalEmech<1/α∧diversity≤τdiv∧Esig≥1/α,global\-prevalenceEmech<1/α∧diversity≤τdiv∧Esig<1/α,rejectotherwise\.\\mathrm\{verdict\}\(H\)=\\begin\{cases\}\\textit\{unit\-specific\}&E^\{\\mathrm\{mech\}\}\\geq 1/\\alpha\\ \\wedge\\ E^\{\\mathrm\{sig\}\}\\geq 1/\\alpha,\\\\\[3\.0pt\] \\textit\{glocal\}&E^\{\\mathrm\{mech\}\}<1/\\alpha\\ \\wedge\\ \\mathrm\{diversity\}\\leq\\tau\_\{\\mathrm\{div\}\}\\ \\wedge\\ E^\{\\mathrm\{sig\}\}\\geq 1/\\alpha,\\\\\[3\.0pt\] \\textit\{global\-prevalence\}&E^\{\\mathrm\{mech\}\}<1/\\alpha\\ \\wedge\\ \\mathrm\{diversity\}\\leq\\tau\_\{\\mathrm\{div\}\}\\ \\wedge\\ E^\{\\mathrm\{sig\}\}<1/\\alpha,\\\\\[3\.0pt\] \\textit\{reject\}&\\text\{otherwise\.\}\\end\{cases\}\(7\)The two*certify*verdicts carry a level\-α\\alphaguarantee—viaEsigE^\{\\mathrm\{sig\}\}for the*glocal*claim \([Section3\.6](https://arxiv.org/html/2608.06621#S3.SS6)\) and the intersection–union bound \([Theorem1](https://arxiv.org/html/2608.06621#Thmtheorem1)\) for the*unit\-specific*claim\.*Global\-prevalence*is a flag rather than a certificate—a constant restriction at the base rate, returned for the user to judge against an external reference \([Section3\.6](https://arxiv.org/html/2608.06621#S3.SS6)\)—while the*reject*branch includes thestructural confound, a diverse hypothesis with a real lift that fails the mechanism CRT \([Theorem2](https://arxiv.org/html/2608.06621#Thmtheorem2)\)\.[Table2](https://arxiv.org/html/2608.06621#S3.T2)enumerates every\(Esig,Emech,diversity\)\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\},\\mathrm\{diversity\}\)combination with its verdict\.

Table 2:Every value combination of\(Esig,Emech,diversity\)\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\},\\mathrm\{diversity\}\)with its verdict\. Significance gates realness \(Esig≈1⇒E^\{\\mathrm\{sig\}\}\\approx 1\\Rightarrownot certified—*reject*, or a*global\-prevalence*flag when the restriction is constant\); among real hypotheses, diversity says whether the hypothesis*claims*per\-unit structure andEmechE^\{\\mathrm\{mech\}\}says whether that claim passes the mechanism CRT\. A diverse claim that fails the CRT—the structural confound—is*rejected*, not certified\. Two combinations cannot occur:Emech≫1E^\{\\mathrm\{mech\}\}\\gg 1forces diversity\>0\>0\.Concrete recommendation\-domain instances of these rows are the planted hypotheses of[Section5](https://arxiv.org/html/2608.06621#S5)\([Table3](https://arxiv.org/html/2608.06621#S5.T3)\); the remaining*sub\-background*row is a degenerate corner with no natural instance, rejected on significance regardless\.

### 3\.6Unit\-invariant hypotheses: glocal and global\-prevalence

Aunit\-invarianthypothesis is one whoseapply​\(x\)\\mathrm\{apply\}\(x\)is*constant over its scope*\. When the scope is a strict, rare subpopulation \(\|U\|<α​\|T\|\|U\|<\\alpha\|T\|\), its significance \(Esig≈\|T\|/\|U\|≫1E^\{\\mathrm\{sig\}\}\\approx\|T\|/\|U\|\\gg 1\) can be certified—we call it*glocal*: global only in the*local*scope\. As the scope grows toward all ofTT,Esig→1E^\{\\mathrm\{sig\}\}\\to 1and the effect becomes internally indistinguishable from the base rate—we call it*global\-prevalence*and return it as a flag to judge against an external reference \([Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)\)\. An example can be: for every phone\-buyer,apply\\mathrm\{apply\}returns the same restriction “buys a case\.” Then every row ofMMis identical, each row\-mean equals the grand mean, and

∵r=ra=rb∀a,b,∴Ejmech=r​\(yj\)1\|U\|​∑k∈Ur​\(yk\)→1\|U\|∑jEjmech=1,diversity=0,\\because r=r\_\{a\}=r\_\{b\}\\forall a,b,\\\>\\therefore E\_\{j\}^\{\\mathrm\{mech\}\}=\\frac\{r\(y\_\{j\}\)\}\{\\frac\{1\}\{\|U\|\}\\sum\_\{k\\in U\}r\(y\_\{k\}\)\}\\rightarrow\\frac\{1\}\{\|U\|\}\\sum\_\{j\}E\_\{j\}^\{\\mathrm\{mech\}\}=1,\\\>\\mathrm\{diversity\}=0,\(8\)whileEsigE^\{\\mathrm\{sig\}\}still fires if phone\-buyers really do buy cases above the population base rate: the pattern is admitted, with no special\-casing, as real but unit\-invariant\. If the restriction*matched to the specific phone*—an iPhone case for an iPhone, a Galaxy case for a Galaxy—the restriction would vary within scope, it is a unit\-specific hypothesis withEmech≫1E^\{\\mathrm\{mech\}\}\\gg 1\.

### 3\.7Efficiency and banking

apply​\(x\)\\mathrm\{apply\}\(x\)—the high computation step—runs once per in\-scope unit \(O​\(\|U\|\)O\(\|U\|\)calls\), notO​\(\|U\|2\)O\(\|U\|^\{2\}\)as a naïve per\-pairing implementation would\. The rest is low computation scoring: the matrix \(mechanism and diversity\) computing complexityO​\(\|U\|2\)O\(\|U\|^\{2\}\), while the significance baseline scores each restriction over all ofTT, anO​\(\|U\|​\|T\|\)O\(\|U\|\\,\|T\|\)term\. SinceT⊇UT\\supseteq Uthis dominates, so the procedure isO​\(\|U\|​\|T\|\)O\(\|U\|\\,\|T\|\)overall\. When\|T\|≫\|U\|\|T\|\\gg\|U\|, replacingTTwith a fixed background subsample of sizeO​\(\|U\|\)O\(\|U\|\)—a valid, slightly noisier null—reduces the cheap scoring toO​\(\|U\|2\)O\(\|U\|^\{2\}\)\.

When a hypothesis is too rare to resolve in one bank \(too few in\-scope*support*units\), draw further independent banks and multiply their per\-bank e\-values \(Esig,\(1:m\)=∏bEsig,\(b\)E^\{\\mathrm\{sig\},\(1:m\)\}=\\prod\_\{b\}E^\{\\mathrm\{sig\},\(b\)\}, and likewiseEmechE^\{\\mathrm\{mech\}\}\), appending banks until a support target ofKKin\-scope units is reached\. This is a direct application of anytime e\-value\(Ramdaset al\.,[2023](https://arxiv.org/html/2608.06621#bib.bib1); Grünwaldet al\.,[2024](https://arxiv.org/html/2608.06621#bib.bib2)\)\.[Algorithm2](https://arxiv.org/html/2608.06621#alg2)states the loop\.

Algorithm 2Bank extension: accumulate support units, then certify \(anytime\-valid\)1:hypothesis

HH, stream of independent banks

B1,B2,…B\_\{1\},B\_\{2\},\\dots, level

α\\alpha, support target

KK, diversity threshold

τdiv\\tau\_\{\\mathrm\{div\}\}
2:

Psig←1P\_\{\\mathrm\{sig\}\}\\leftarrow 1,

Pmech←1P\_\{\\mathrm\{mech\}\}\\leftarrow 1,

D←0D\\leftarrow 0,

n←0n\\leftarrow 0,

m←0m\\leftarrow 0⊳\\trianglerightproducts; diversity mass; support

3:while

n<Kn<Kanda bank remainsdo⊳\\trianglerightextend untilKKin\-scope units collected

4:

m←m\+1m\\leftarrow m\+1; open

BmB\_\{m\};

Um←\{i∈Bm:in​\_​scope​\(xi\)=1\}U\_\{m\}\\leftarrow\\\{i\\in B\_\{m\}:\\mathrm\{in\\\_scope\}\(x\_\{i\}\)=1\\\};

n←n\+\|Um\|n\\leftarrow n\+\|U\_\{m\}\|
5:

\(Esig,\(m\),Emech,\(m\),d\(m\)\)←\(E^\{\\mathrm\{sig\},\(m\)\},E^\{\\mathrm\{mech\},\(m\)\},d^\{\(m\)\}\)\\leftarrowsignals of

HHon

BmB\_\{m\}⊳\\triangleright[Algorithm1](https://arxiv.org/html/2608.06621#alg1)onBmB\_\{m\}

6:

Psig←Psig⋅Esig,\(m\)P\_\{\\mathrm\{sig\}\}\\leftarrow P\_\{\\mathrm\{sig\}\}\\cdot E^\{\\mathrm\{sig\},\(m\)\},

Pmech←Pmech⋅Emech,\(m\)P\_\{\\mathrm\{mech\}\}\\leftarrow P\_\{\\mathrm\{mech\}\}\\cdot E^\{\\mathrm\{mech\},\(m\)\},

D←D\+\|Um\|​d\(m\)D\\leftarrow D\+\|U\_\{m\}\|\\,d^\{\(m\)\}
7:endwhile

8:if

n<Kn<Kthen

9:returnundecided⊳\\trianglerightstream exhausted before reaching supportKK

10:endif

11:return

verdict​\(Psig,Pmech,D/n\)\\mathrm\{verdict\}\(P\_\{\\mathrm\{sig\}\},P\_\{\\mathrm\{mech\}\},D/n\)per[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)⊳\\trianglerightunit\-specific / glocal / global\-prevalence / reject

## 4Theory

Throughout, each restrictionr​\(⋅\)r\(\\cdot\)is a nonnegative score and a function of the input alone—hence independent of the outcomes it is scored against under either null\.

###### Proposition 1\(Significance validity\)\.

EjsigE\_\{j\}^\{\\mathrm\{sig\}\}is an exact e\-value for the exchangeability nullH0sigH\_\{0\}^\{\\mathrm\{sig\}\}: under a*random*\(no\-lift\) hypothesis𝔼​\[Ejsig\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{sig\}\}\]=1, so only genuine lift over the base rate drives it above11\.

###### Proof\.

Fix a unitj∈Uj\\in Uand condition on the outcome pool\{yk\}k∈T\\\{y\_\{k\}\\\}\_\{k\\in T\}\(as a multiset\) together withrjr\_\{j\}\. Sincerj=apply​\(xj\)r\_\{j\}=\\mathrm\{apply\}\(x\_\{j\}\)is a function of the input alone, it is fixed under this conditioning and does not depend on the outcomes\. The denominatorg0\(rj\)=1\|T\|​∑k∈Trj​\(yk\)g\_\{0\}^\{\(r\_\{j\}\)\}=\\tfrac\{1\}\{\|T\|\}\\sum\_\{k\\in T\}r\_\{j\}\(y\_\{k\}\)is a symmetric function of the pool, hence a constant given the conditioning\.*Exchangeability*ofyjy\_\{j\}with the pool underH0sigH\_\{0\}^\{\\mathrm\{sig\}\}means that, conditionally on the pool,yjy\_\{j\}is equally likely to be any of its\|T\|\|T\|members—a uniform draw—so its conditional mean equals the pool average:

𝔼​\[rj​\(yj\)∣pool\]=1\|T\|​∑k∈Trj​\(yk\)=g0\(rj\)\.\\mathbb\{E\}\\\!\\left\[r\_\{j\}\(y\_\{j\}\)\\mid\\text\{pool\}\\right\]=\\frac\{1\}\{\|T\|\}\\sum\_\{k\\in T\}r\_\{j\}\(y\_\{k\}\)=g\_\{0\}^\{\(r\_\{j\}\)\}\.Dividing by the constantg0\(rj\)g\_\{0\}^\{\(r\_\{j\}\)\}gives𝔼​\[Ejsig∣pool\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{sig\}\}\\mid\\text\{pool\}\]=1, and by the tower property𝔼​\[Ejsig\]=𝔼​\[𝔼​\[Ejsig∣pool\]\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{sig\}\}\]=\\mathbb\{E\}\[\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{sig\}\}\\mid\\text\{pool\}\]\]=1\. The identity is*exact*\(equal to11, not merely≤1\\leq 1\) becausej∈Tj\\in T: the pool definingg0\(rj\)g\_\{0\}^\{\(r\_\{j\}\)\}already containsyjy\_\{j\}, the conformal self\-inclusion that removes any off\-by\-one bias\. ∎

###### Proposition 2\(Mechanism validity, conformal\)\.

EjmechE\_\{j\}^\{\\mathrm\{mech\}\}is an exact e\-value for the exchangeability nullH0mechH\_\{0\}^\{\\mathrm\{mech\}\}: under a*random*\(no\-mechanism\) hypothesis𝔼​\[Ejmech\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{mech\}\}\]=1, so only genuine per\-unit structure drives it above11\.

###### Proof\.

Identical recipe with the in\-scope pool\. Fixj∈Uj\\in Uand condition on\{yk\}k∈U\\\{y\_\{k\}\\\}\_\{k\\in U\}and onrjr\_\{j\}\. The denominator, the row mean1\|U\|​∑k∈Urj​\(yk\)\\tfrac\{1\}\{\|U\|\}\\sum\_\{k\\in U\}r\_\{j\}\(y\_\{k\}\), is a symmetric function of the pool, hence constant given the conditioning—in particular invariant to the within\-scope pairing thatH0mechH\_\{0\}^\{\\mathrm\{mech\}\}randomizes\. UnderH0mechH\_\{0\}^\{\\mathrm\{mech\}\}that pairing is exchangeable, soyjy\_\{j\}is conditionally a uniform draw from the in\-scope pool, and

𝔼​\[M​\[j\]​\[j\]∣pool\]=𝔼​\[rj​\(yj\)∣pool\]=1\|U\|​∑k∈Urj​\(yk\),\\mathbb\{E\}\\\!\\left\[M\[j\]\[j\]\\mid\\text\{pool\}\\right\]=\\mathbb\{E\}\\\!\\left\[r\_\{j\}\(y\_\{j\}\)\\mid\\text\{pool\}\\right\]=\\frac\{1\}\{\|U\|\}\\sum\_\{k\\in U\}r\_\{j\}\(y\_\{k\}\),which is exactly the denominator\. Hence𝔼​\[Ejmech∣pool\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{mech\}\}\\mid\\text\{pool\}\]=1, and by the tower property𝔼​\[Ejmech\]=1\\mathbb\{E\}\[E\_\{j\}^\{\\mathrm\{mech\}\}\]=1\. ∎

###### Proposition 3\(Dependence\-robust aggregation\)\.

The mean of e\-values is an e\-value under arbitrary dependence; henceEsigE^\{\\mathrm\{sig\}\}andEmechE^\{\\mathrm\{mech\}\}are valid despite inter\-unit dependence from the shared pool\.

###### Proof\.

EsigE^\{\\mathrm\{sig\}\}andEmechE^\{\\mathrm\{mech\}\}are*means*of the per\-unit e\-values\. By linearity of expectation—which holds under*arbitrary*dependence—

𝔼​\[1\|U\|​∑j∈UEj\]=1\|U\|​∑j∈U𝔼​\[Ej\]≤1\|U\|​∑j∈U1=1,\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}E\_\{j\}\\right\]=\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}\\mathbb\{E\}\[E\_\{j\}\]\\leq\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}1=1,using𝔼​\[Ej\]≤1\\mathbb\{E\}\[E\_\{j\}\]\\leq 1from[Propositions1](https://arxiv.org/html/2608.06621#Thmproposition1)and[2](https://arxiv.org/html/2608.06621#Thmproposition2)\. The per\-unit e\-values are dependent—they are read off the shared outcome pool—but linearity never invokes independence, so the mean is a valid e\-value regardless\. \(A*product*would instead require independence, which is why the mean, not the product, aggregates within a pool; independent*banks*are combined by a product in[Section3\.7](https://arxiv.org/html/2608.06621#S3.SS7)\.\) ∎

###### Theorem 1\(Certification error\)\.

Under “certify iffEsig≥1/αE^\{\\mathrm\{sig\}\}\\geq 1/\\alphaandEmech≥1/αE^\{\\mathrm\{mech\}\}\\geq 1/\\alpha,” the per\-hypothesis certification error is≤α\\leq\\alpha, with no multiplicity correction\.

###### Proof\.

“Certify” requiresEsig≥1/αE^\{\\mathrm\{sig\}\}\\geq 1/\\alpha*and*Emech≥1/αE^\{\\mathrm\{mech\}\}\\geq 1/\\alpha\. A hypothesis fails to be real\-and\-unit\-specific exactly when at least one component null holds, i\.e\. under the composite nullH0sig∪H0mechH\_\{0\}^\{\\mathrm\{sig\}\}\\cup H\_\{0\}^\{\\mathrm\{mech\}\}\. IfH0sigH\_\{0\}^\{\\mathrm\{sig\}\}holds,EsigE^\{\\mathrm\{sig\}\}is a valid e\-value there \([Propositions1](https://arxiv.org/html/2608.06621#Thmproposition1)and[3](https://arxiv.org/html/2608.06621#Thmproposition3)\), so Markov’s inequality givesℙ​\(Esig≥1/α\)≤α\\mathbb\{P\}\(E^\{\\mathrm\{sig\}\}\\geq 1/\\alpha\)\\leq\\alpha; since certification requiresEsig≥1/αE^\{\\mathrm\{sig\}\}\\geq 1/\\alpha,ℙ​\(certify\)≤α\\mathbb\{P\}\(\\text\{certify\}\)\\leq\\alpha\. The argument underH0mechH\_\{0\}^\{\\mathrm\{mech\}\}viaEmechE^\{\\mathrm\{mech\}\}is symmetric\. Thus the certification error is≤α\\leq\\alphaunder either component—the*intersection–union*principle: demanding that*both*level\-α\\alphae\-value tests fire yields overall levelα\\alphawith no multiplicity correction\. ∎

###### Theorem 2\(Diversity resolves glocal vs\. confound\)\.

WhenEsig≫1E^\{\\mathrm\{sig\}\}\\gg 1andEmech≈1E^\{\\mathrm\{mech\}\}\\approx 1, a glocal hypothesis and a structural confound induce the same law of\(Esig,Emech\)\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\}\), so no test on that pair separates them; they differ only in diversity\. A glocal hypothesis has, by definition, a restriction constant over its scope, so every column ofMMis constant, givingσj=0\\sigma\_\{j\}=0for alljjandD=0D=0; a structural confound has restrictions that vary across units, so—provided the observed outcomes distinguish them—some column varies andD\>0D\>0\. Hence, under that condition,

D=0⇔glocal,D\>0⇔structural confound,D=0\\iff\\text\{glocal\},\\qquad D\>0\\iff\\text\{structural confound\},\(9\)a separation requiring no distributional assumption, only that the observed outcomes distinguish distinct restrictions \([AppendixB](https://arxiv.org/html/2608.06621#A2)\)\.

###### Proof\.

WriteM¯j=1\|U\|​∑k∈UM​\[j\]​\[k\]\\bar\{M\}\_\{j\}=\\tfrac\{1\}\{\|U\|\}\\sum\_\{k\\in U\}M\[j\]\[k\]\.*Same law\.*For a glocal hypothesis one restrictionrris shared acrossUU, soM​\[i\]​\[j\]=r​\(yj\)M\[i\]\[j\]=r\(y\_\{j\}\); every row is equal,M¯j≡r¯\\bar\{M\}\_\{j\}\\equiv\\overline\{r\}, henceEmech=1E^\{\\mathrm\{mech\}\}=1, whileEsig≫1E^\{\\mathrm\{sig\}\}\\gg 1ifrrbeats the base rate\. A confound has per\-unit restrictions each aligned with a shared exogenous factor—ri​\(yj\)r\_\{i\}\(y\_\{j\}\)depends onyjy\_\{j\}only through that factor, not thei↔ji\\\!\\leftrightarrow\\\!jpairing—calibrated to the same diagonal and row\-mean marginals, so𝔼​\[M​\[j\]​\[j\]\]=𝔼​\[M¯j\]\\mathbb\{E\}\[M\[j\]\[j\]\]=\\mathbb\{E\}\[\\bar\{M\}\_\{j\}\]givesEmech=1E^\{\\mathrm\{mech\}\}=1and the sameEsigE^\{\\mathrm\{sig\}\}; the two share the law of\(Esig,Emech\)\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\}\)\.*Diversity separates them\.*If the hypothesis is glocal,apply\\mathrm\{apply\}is constant overUU:ri≡rr\_\{i\}\\equiv r, soM​\[i\]​\[j\]=r​\(yj\)M\[i\]\[j\]=r\(y\_\{j\}\)does not depend onii; each column is constant, givingσj=0\\sigma\_\{j\}=0for everyjjandD=1\|U\|​∑j∈Uσj/μj=0D=\\tfrac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}\\sigma\_\{j\}/\\mu\_\{j\}=0\. If it is a structural confound, therir\_\{i\}are not all equal, so—provided the observed outcomes distinguish them—someyjy\_\{j\}has\{ri​\(yj\)\}i∈U\\\{r\_\{i\}\(y\_\{j\}\)\\\}\_\{i\\in U\}not all equal, givingσj\>0\\sigma\_\{j\}\>0andD\>0D\>0\. The two cases are mutually exclusive and exhaust theEsig≫1,Emech≈1E^\{\\mathrm\{sig\}\}\\gg 1,E^\{\\mathrm\{mech\}\}\\approx 1region, soDDseparates glocal from confound whenever the observed outcomes distinguish distinct restrictions\. ∎

[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)applies this through the toleranceτdiv\\tau\_\{\\mathrm\{div\}\}rather than the exact zero above, so restrictions that are similar without being identical—differing slightly, or on few outcomes—are tolerated as constant \([AppendixB](https://arxiv.org/html/2608.06621#A2)\)\. Considering a restriction can be implemented in any form which can even be non\-deterministic, these result comparison seems the only universal way to compare restrictions\. And due to the score noises \([SectionB\.6](https://arxiv.org/html/2608.06621#A2.SS6)\), thisτdiv\\tau\_\{\\mathrm\{div\}\}tolerance is necessary\.

## 5Experiments

Real data cannot validate a*certifier*since the hypothesis correctness is unknown\. Therefore a synthetic ads\-recommendation world with*planted*ground truth—several real rules and several fakes, one per threat the method must catch—is constructed and each verdict is checked against the known class\. This isolates “does the certifier assign the right verdict?” from any modeling of real behavior\. All numbers are means over1010random worlds; the generator and a pure\-Python reference certifier are described in[AppendixA](https://arxiv.org/html/2608.06621#A1)\.

Table 3:Per\-rule certification for each method \(right \(✓\\checkmark\) or wrong \(×\\times\)\) where the baseline is naive held\-out\.Table 4:The eight planted hypotheses—four real, four fakeExperimentation details are shown in[AppendixA](https://arxiv.org/html/2608.06621#A1)\. The hypothesis details are in[Table4](https://arxiv.org/html/2608.06621#S5.T4)\. In[Table3](https://arxiv.org/html/2608.06621#S5.T3), it can be seen that ourN×NN\\times Ne\-valuation did not certify any fake, while the baseline \(naive held\-out data test\) was fooled by all four fakes\. It is worth emphasizing that the mechanism e\-value is what rejected the structural confound\.

## 6Conclusion

We presented a domain\-general, anytime\-valid certifier for open\-ended machine\-proposed hypotheses: a population\-baseline conformal significance e\-value and an in\-scope conformal mechanism e\-value from anN×NN\\times Ncross\-prediction matrix, combined by an intersection–union test with a proved bound, composable across banks and rounds, with a unifying\(Esig,Emech,diversity\)\(E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\},\\text\{diversity\}\)classification that subsumes unit\-invariant \(glocal and global\-prevalence\) associations and an identifiability theorem delimiting what the matrix can decide\. On synthetic data with planted ground truth it recovers the correct verdict where held\-out testing is fooled by a structural confound\. It is a recipe for certifying machine\-proposed hypotheses under a strict “the machine proposes, data certifies” contract; reasoning\-pattern discovery for recommendation is one instantiation\.

## References

- A survey on hypothesis generation for scientific discovery in the era of large language models\.arXiv preprint arXiv:2504\.05496\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p2.1)\.
- A\. Amayuelas, J\. Sain, S\. Kaur, and C\. Smiley \(2025\)Grounding LLM reasoning with knowledge graphs\.arXiv preprint arXiv:2502\.13247\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- E\. Candès, Y\. Fan, L\. Janson, and J\. Lv \(2018\)Panning for gold: model\-x knockoffs for high\-dimensional controlled variable selection\.Journal of the Royal Statistical Society: Series B80\(3\),pp\. 551–577\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1),[§1](https://arxiv.org/html/2608.06621#S1.p9.5)\.
- A\. Fallahpour, A\. Magnuson, P\. Gupta, S\. Ma,et al\.\(2025\)BioReason: incentivizing multimodal biological reasoning within a DNA\-LLM model\.arXiv preprint arXiv:2505\.23579\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)PAL: program\-aided language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. Wichmann \(2020\)Shortcut learning in deep neural networks\.Nature Machine Intelligence2\(11\),pp\. 665–673\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1)\.
- S\. Geng, M\. Josifoski, M\. Peyrard, and R\. West \(2023\)Grammar\-constrained decoding for structured NLP tasks without finetuning\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p5.1)\.
- Google DeepMind \(2025\)Olympiad\-level formal mathematical reasoning with reinforcement learning\.Nature\.Note:[https://www\.nature\.com/articles/s41586\-025\-09833\-y](https://www.nature.com/articles/s41586-025-09833-y); individual author list to be finalizedCited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- P\. Grünwald, R\. de Heide, and W\. M\. Koolen \(2024\)Safe testing\.Journal of the Royal Statistical Society: Series B86\(5\),pp\. 1091–1128\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p9.5),[§3\.7](https://arxiv.org/html/2608.06621#S3.SS7.p2.3),[§3](https://arxiv.org/html/2608.06621#S3.p1.7)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2024\)Large language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p3.1)\.
- R\. Kamoi, Y\. Zhang, N\. Zhang, J\. Han, and R\. Zhang \(2024\)When can LLMs actually correct their own mistakes? a critical survey of self\-correction of LLMs\.Transactions of the Association for Computational Linguistics \(TACL\)\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p3.1)\.
- H\. Kang, J\. Ni, and H\. Yao \(2023\)EVER: mitigating hallucination in large language models through real\-time verification and rectification\.arXiv preprint arXiv:2311\.09114\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p3.1)\.
- B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. Dalvi Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. Clark \(2025\)DiscoveryBench: towards data\-driven discovery with large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1)\.
- J\. Markovic, L\. Xia, and J\. Taylor \(2017\)A unifying approach to selective inference with applications to cross\-validation\.arXiv preprint arXiv:1703\.06559\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1)\.
- OneRec Team \(2026\)OneReason technical report\.Technical reportKuaishou\.Note:arXiv:2606\.06260Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p5.1)\.
- A\. Ramdas, P\. Grünwald, V\. Vovk, and G\. Shafer \(2023\)Game\-theoretic statistics and safe anytime\-valid inference\.Statistical Science38\(4\),pp\. 576–601\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p9.5),[§3\.7](https://arxiv.org/html/2608.06621#S3.SS7.p2.3),[§3](https://arxiv.org/html/2608.06621#S3.p1.7)\.
- S\. Shaer, G\. Maman, and Y\. Romano \(2023\)Model\-X sequential testing for conditional independence via testing by betting\.InInternational Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1)\.
- P\. Song, K\. Yang, and A\. Anandkumar \(2025\)Lean copilot: large language models as copilots for theorem proving in lean\.InInternational Conference on Neuro\-symbolic Systems \(NeuS\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- V\. Vovk and R\. Wang \(2021\)E\-values: calibration, combination, and applications\.The Annals of Statistics49\(3\),pp\. 1736–1754\.Cited by:[§3\.3](https://arxiv.org/html/2608.06621#S3.SS3.p1.4)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.
- W\. Ye, L\. Jiang, E\. Xie, G\. Zheng, Y\. Ma, X\. Cao,et al\.\(2024\)The clever hans mirage: a comprehensive survey on spurious correlations in machine learning\.arXiv preprint arXiv:2402\.12715\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p6.1)\.
- L\. Zhang, A\. Hosseini, H\. Bansal, M\. Kazemi, A\. Kumar, and R\. Agarwal \(2025\)Generative verifiers: reward modeling as next\-token prediction\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p3.1)\.
- Y\. Zhao, Z\. Liu, Y\. Zheng, and K\. Lam \(2026\)Attribution techniques for mitigating hallucinated information in RAG systems: a survey\.arXiv preprint arXiv:2601\.19927\.Cited by:[§1](https://arxiv.org/html/2608.06621#S1.p4.1)\.

## Appendix AExperiment details

Generator\.A catalog of15001500items, each with a category \(of4040\), a brand \(of6060\), and a Zipf popularity \(s=1\.05s\{=\}1\.05\)\. Each of\|T\|=5000\|T\|\{=\}5000users draws a scope from\{complement, brand, category, holiday, exposed, base\}\\\{\\text\{complement, brand, category, holiday, exposed, base\}\\\}with weights\(0\.03,0\.03,0\.03,0\.03,0\.03,0\.85\)\(0\.03,0\.03,0\.03,0\.03,0\.03,0\.85\)\(so each rule scope is≈3%\{\\approx\}3\\%,\|U\|≈150\|U\|\{\\approx\}150, and\|T\|/\|U\|≈33\>1/α\|T\|/\|U\|\{\\approx\}33\>1/\\alpha\); a historyxix\_\{i\}\(1515items mixing the user’s brand/category/recent item with popularity draws\); and a held\-out outcome setyiy\_\{i\}\(66items\) generated from the scope’s mechanism with probabilitypfollow=0\.9p\_\{\\mathrm\{follow\}\}\{=\}0\.9, else popularity noise\. The structural confound is planted separately: each “segment” covers a different∼60%\{\\sim\}60\\%subset of a*low\-popularity*promoted setPP\(soEsigE^\{\\mathrm\{sig\}\}is high\), exposed users’yi⊂Py\_\{i\}\\subset P, and the segment→\\tooutcome score factors as \(coverage\)×\\times\(item appeal\) with no pairing term, givingEmech≈1E^\{\\mathrm\{mech\}\}\\approx 1with diversity\>0\{\>\}0\.

Certifier\.ri​\(y\)=r\_\{i\}\(y\)=recall of restrictioniion outcome setyy\(\|retainedi∩y\|/\|y\|\|\\text\{retained\}\_\{i\}\\cap y\|/\|y\|\);Esig,Emech,E^\{\\mathrm\{sig\}\},E^\{\\mathrm\{mech\}\},and diversity exactly as in[Section3](https://arxiv.org/html/2608.06621#S3), withg0\(ri\)g\_\{0\}^\{\(r\_\{i\}\)\}estimated over a subsample ofTT\(size20002000\) and the matrix capped at\|U\|=400\|U\|\{=\}400\.τdiv\\tau\_\{\\mathrm\{div\}\}is calibrated per world as the midpoint between a known glocal \(diversity0\) and a known confound;α=0\.05\\alpha\{=\}0\.05; single evaluation \(no banks\)\. An optional per\-user check flags a rule heterogeneous when fewer than half its in\-scope units satisfyri​\(yi\)\>g0\(ri\)r\_\{i\}\(y\_\{i\}\)\>g\_\{0\}^\{\(r\_\{i\}\)\}\.

Baseline\.Naive held\-out predictive validation: a one\-sidedzz\-test that the rule’s mean held\-out recall exceeds that of a random same\-size set \(predicts above chance\)\. It certifies anything predictive—all four fakes included\.

Protocol\.1010seeds for the per\-rule table \([Table3](https://arxiv.org/html/2608.06621#S5.T3)\)\.

## Appendix BThe diversity statistic: whatDDresolves

[Section3\.4](https://arxiv.org/html/2608.06621#S3.SS4)definesDDas the mean column coefficient of variation ofMMand[Theorem2](https://arxiv.org/html/2608.06621#Thmtheorem2)shows it separates a glocal hypothesis from a structural confound whenever the observed outcomes distinguish distinct restrictions\. This appendix records what that condition excludes, and what the magnitude ofDDdoes and does not encode\.

### B\.1What “the same restriction” means here

A glocal hypothesis is one whoseapply\\mathrm\{apply\}is constant over its scope: therir\_\{i\}are one function\. That is a claim of*identity*, and for an arbitraryapply\\mathrm\{apply\}it cannot be tested directly—two procedures may compute the same function, and two different functions may agree on any finite sample\. Identity can therefore only be*probed*, and probing yields a*similarity*rather than identity:rir\_\{i\}andrkr\_\{k\}count as the same when they score alike on whatever is probed\.

Any such similarity is fixed by three choices: theprobe seton which the restrictions are compared, themagnitudeby which a difference at a probe point is measured, and thepoolingof those differences across the probe and across units\.DDis one particular choice of all three—probe set==the observed outcomes\{yj\}j∈U\\\{y\_\{j\}\\\}\_\{j\\in U\}, magnitude==the within\-column coefficient of variation, pooling==the mean over columns\. The notion of sameness it implements is thus*outcome\-weighted interchangeability*: two restrictions are the same to the extent that exchanging one for the other leaves predictions unchanged on the outcome distribution\. For a certifier whose subject is prediction this is a natural weighting, since a difference then counts in proportion to how often it affects an outcome that is actually observed\.

Read this way, the condition in[Theorem2](https://arxiv.org/html/2608.06621#Thmtheorem2)is the statement that on this probe similarity coincides with identity—which is why the forward direction needs no assumption and the converse does—and the routing toleranceτdiv\\tau\_\{\\mathrm\{div\}\}of[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)sets how much dissimilarity still counts as sameness\. The remainder of this appendix records what follows from the three choices: the probe may be too coarse \([SectionB\.3](https://arxiv.org/html/2608.06621#A2.SS3)\), the pooling trades prevalence against magnitude \([SectionB\.4](https://arxiv.org/html/2608.06621#A2.SS4)\), an empty probe response calls for a different bound \([SectionB\.5](https://arxiv.org/html/2608.06621#A2.SS5)\), and the similarity itself can carry an error bar \([SectionB\.7](https://arxiv.org/html/2608.06621#A2.SS7)\)\.

### B\.2An exact factorization

Splitting the sum by whether a column distinguishes the restrictions, withS=\{j∈U:σj\>0\}S=\\\{j\\in U:\\sigma\_\{j\}\>0\\\},

D=1\|U\|​∑j∈Uσjμj=\|S\|\|U\|⏟δ^⋅1\|S\|​∑j∈Sσjμj⏟c,D=\\frac\{1\}\{\|U\|\}\\sum\_\{j\\in U\}\\frac\{\\sigma\_\{j\}\}\{\\mu\_\{j\}\}=\\underbrace\{\\frac\{\|S\|\}\{\|U\|\}\}\_\{\\hat\{\\delta\}\}\\;\\cdot\\;\\underbrace\{\\frac\{1\}\{\|S\|\}\\sum\_\{j\\in S\}\\frac\{\\sigma\_\{j\}\}\{\\mu\_\{j\}\}\}\_\{c\},\(10\)an identity, not an approximation:δ^\\hat\{\\delta\}is the fraction of outcomes that exercise a difference between the restrictions andccis the mean coefficient of variation among those columns\. One level further, for a column in which a fractionqqof units deviate byΔ\\Deltafrom the remainder,μ=\(1−q\)​v\+q​\(v−Δ\)\\mu=\(1\-q\)v\+q\(v\-\\Delta\)andσ=q​\(1−q\)​\|Δ\|\\sigma=\\sqrt\{q\(1\-q\)\}\\,\|\\Delta\|, so

c=q​\(1−q\)​\|Δ\|μ,henceD≈δ⏟how often⋅q​\(1−q\)⏟how many units⋅\|Δ\|/μ⏟how much\.c=\\sqrt\{q\(1\-q\)\}\\;\\frac\{\|\\Delta\|\}\{\\mu\},\\qquad\\text\{hence\}\\qquad D\\;\\approx\\;\\underbrace\{\\delta\}\_\{\\text\{how often\}\}\\cdot\\underbrace\{\\sqrt\{q\(1\-q\)\}\}\_\{\\text\{how many units\}\}\\cdot\\underbrace\{\|\\Delta\|/\\mu\}\_\{\\text\{how much\}\}\.\(11\)DDtherefore compounds three distinct quantities, and a given value does not identify them: a rare but sharp divergence and a common but slight one can produce the sameDD\. Reportingδ^\\hat\{\\delta\}andccalongsideDDcosts nothing, since both are read off the pass that already formsDD\.

### B\.3What the separating\-outcome condition excludes

The condition in[Theorem2](https://arxiv.org/html/2608.06621#Thmtheorem2)is a statement about the sample, not about the restrictions: distinct functions may agree on a finite set\. Concretely, let two per\-unit restrictions share a common element and differ only on items absent from\{yj\}j∈U\\\{y\_\{j\}\\\}\_\{j\\in U\}\. Every entry of every column is then equal, soσj=0\\sigma\_\{j\}=0for alljjandD=0D=0, whileapply\\mathrm\{apply\}is not constant overUU\. Such a hypothesis lies inside theEsig≫1,Emech≈1E^\{\\mathrm\{sig\}\}\\gg 1,E^\{\\mathrm\{mech\}\}\\approx 1region and would be read as glocal\. The condition is what rules this out, and it is a genuine assumption rather than a consequence of the restrictions differing\.

Two remarks bound its cost\. First, if the restrictions diverge on outcome massδ\>0\\delta\>0, the probability that no observed column distinguishes them is\(1−δ\)\|U\|\(1\-\\delta\)^\{\|U\|\}, so the condition holds with probability approaching one as the in\-scope support grows\. Second, nothing forces the probe to be the observed outcomes:DDis not an e\-value and respects no null, so the same statistic may be evaluated on any probe setP⊇\{yj\}j∈UP\\supseteq\\\{y\_\{j\}\\\}\_\{j\\in U\}—the item universe, for instance—at a cost ofO​\(\|U\|​\|P\|\)O\(\|U\|\\,\|P\|\)cheap scoring calls, the same order as the significance term already paid\. WideningPPcan only increase what is distinguished\.

### B\.4Behaviour of the magnitude

By[Equation10](https://arxiv.org/html/2608.06621#A2.E10),D→δ⋅cD\\to\\delta\\cdot cas\|U\|\|U\|grows: additional support concentratesDDat that value rather than raising it\. The thresholdτdiv\\tau\_\{\\mathrm\{div\}\}therefore acts on the*product*of prevalence and severity, and in particular a hypothesis is routed as constant wheneverδ⋅c≤τdiv\\delta\\cdot c\\leq\\tau\_\{\\mathrm\{div\}\}, independently of\|U\|\|U\|\. Two consequences are worth stating\.

*Sparse divergence\.*If the restrictions differ on a small share of outcomes,δ^\\hat\{\\delta\}is small andDDis correspondingly small however sharply they differ on that share\.

*Lopsided divergence\.*By[Equation11](https://arxiv.org/html/2608.06621#A2.E11),c=O​\(q\)c=O\(\\sqrt\{q\}\)for smallqq, so a fixed number of deviant units givesc=O​\(1/\|U\|\)c=O\(1/\\sqrt\{\|U\|\}\): a single unit whose restriction differs completely contributesD≈0\.14D\\approx 0\.14at\|U\|=50\|U\|=50butD≈0\.03D\\approx 0\.03at\|U\|=1000\|U\|=1000\. A vanishing subgroup is thus absorbed as support grows\. For a certificate read as*interchangeability*—the expected cost of treating therir\_\{i\}as one restriction—this is the desired behaviour, since a1/\|U\|1/\|U\|fraction contributes negligibly to that cost\. It is a limitation only for a certificate read as*identity*\.

### B\.5The caseδ^=0\\hat\{\\delta\}=0

When no column distinguishes the restrictions,ccin[Equation10](https://arxiv.org/html/2608.06621#A2.E10)is an average over an empty set and is*unobserved*;D=0D=0then carries no information about how large a divergence could be hiding\. This case admits a distribution\-free bound instead\. Conditional on the restriction set, the per\-column indicators𝟏​\[σj\>0\]\\mathbf\{1\}\[\\sigma\_\{j\}\>0\]are i\.i\.d\. Bernoulli\(δ\)\(\\delta\), soδ^\\hat\{\\delta\}is a binomial proportion and observingδ^=0\\hat\{\\delta\}=0over\|U\|\|U\|units certifies

δ≤1−η1/\|U\|≈3/\|U\|at confidence​1−η\(η=0\.05\),\\delta\\;\\leq\\;1\-\\eta^\{1/\|U\|\}\\;\\approx\\;3/\|U\|\\qquad\\text\{at confidence \}1\-\\eta\\ \\ \(\\eta=0\.05\),\(12\)which is0\.280\.28at\|U\|=9\|U\|=9,0\.0580\.058at\|U\|=50\|U\|=50and0\.00300\.0030at\|U\|=1000\|U\|=1000\. Unlike the magnitude ofDD, this bound tightens with support, so it quantifies how much a constancy reading is worth at a given\|U\|\|U\|\. It also supplies the likelihood the posterior analysis of[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)would otherwise have to assume: a strictly constant hypothesis yieldsδ^=0\\hat\{\\delta\}=0with probability one, while a confound with divergence massδ\\deltadoes so with probability\(1−δ\)\|U\|\(1\-\\delta\)^\{\|U\|\}\.

### B\.6Noise

Two noise sources act differently onDD\.*Label noise*—a perturbed outcomeyjy\_\{j\}—cannot inflate it: every entry of columnjjscores the sameyjy\_\{j\}, so the perturbation is common to the column and cannot create spread within it; with identical restrictionsσj=0\\sigma\_\{j\}=0exactly, whateveryjy\_\{j\}arrives\. The normalizationσj/μj\\sigma\_\{j\}/\\mu\_\{j\}additionally cancels any column\-common rescaling, such as a varying outcome size\|yj\|\|y\_\{j\}\|under the recall score of[AppendixA](https://arxiv.org/html/2608.06621#A1)\.*Score noise*—a stochastic or estimatedri​\(⋅\)r\_\{i\}\(\\cdot\)—is per\-entry and does inflateσj\\sigma\_\{j\}; where it is present, a column should be admitted toSSonly if its spread exceeds a floor obtained by variance decomposition \(evaluateri​\(y\)r\_\{i\}\(y\)twice; the within\-restriction variance is the floor\)\. With deterministic restrictions, as in[AppendixA](https://arxiv.org/html/2608.06621#A1), this floor is zero\.

### B\.7A bounded variant

The routing rule of[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)thresholds a point estimate\. Where an explicit error level is wanted on the routing itself, the binomial bounds of[SectionB\.5](https://arxiv.org/html/2608.06621#A2.SS5)give\[δ^lo,δ^hi\]\[\\hat\{\\delta\}\_\{\\mathrm\{lo\}\},\\hat\{\\delta\}\_\{\\mathrm\{hi\}\}\]and hence\[Dlo,Dhi\]=\[δ^lo​c,δ^hi​c\]\[D\_\{\\mathrm\{lo\}\},D\_\{\\mathrm\{hi\}\}\]=\[\\hat\{\\delta\}\_\{\\mathrm\{lo\}\}c,\\,\\hat\{\\delta\}\_\{\\mathrm\{hi\}\}c\], so one may route as constant whenDhi≤τdivD\_\{\\mathrm\{hi\}\}\\leq\\tau\_\{\\mathrm\{div\}\}, as divergent whenDlo\>τdivD\_\{\\mathrm\{lo\}\}\>\\tau\_\{\\mathrm\{div\}\}, and return*undecided*otherwise—with[Equation12](https://arxiv.org/html/2608.06621#A2.E12)deciding theδ^=0\\hat\{\\delta\}=0case, whereDDcannot be bounded from the data\. This variant is strictly more conservative: it never routes as constant a hypothesis the point\-estimate rule would route as divergent, and it converts an unresolvable case into a request for further support rather than a default\. The experiments in[Section5](https://arxiv.org/html/2608.06621#S5)use the point\-estimate rule as stated in[Section3\.5](https://arxiv.org/html/2608.06621#S3.SS5)\.