划分、提示、聚合:语言模型中的统计自洽性
摘要
该论文提出了一种基于全概率定律的统计自一致性评测框架,通过二叉树分区、提示与聚合来检验LLM条件概率估计的一致性,发现前沿模型普遍存在违反一致性的问题,并揭示了细粒度子群体估计反而更贴近人类参考数据的'宏观谬误'现象。
arXiv:2607.15277v2 Announce Type: replace
Abstract: Large language models (LLMs) are often used to perform a form of conditional inference. In-context learning is one implicit way to do this. The prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Translating these comparisons into a benchmark score and evaluating frontier models across problem domains, we demonstrate widespread violations of basic consistency properties. An in-depth study of persona prompting further reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. The effect persists across variations in partitioning, verbalization, and estimation task, suggesting that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.
查看缓存全文
缓存时间: 2026/11/20 09:13
# Partition, Prompt, Aggregate:Statistical Self-Consistency in Language Models
Source: [https://arxiv.org/html/2607.15277](https://arxiv.org/html/2607.15277)
Patrik WolfAffiliation:Max Planck Institute for Intelligent Systems, Tübingen, GermanyAffiliation:ETH ZürichAffiliation:ELLIS Institute TübingenAffiliation:Tübingen AI CenterAndreas KrauseAffiliation:ETH ZürichCelestine Mendler\-DünnerAffiliation:Max Planck Institute for Intelligent Systems, Tübingen, GermanyAffiliation:ETH ZürichAffiliation:ELLIS Institute TübingenAffiliation:Tübingen AI Center
###### Abstract
Large language models \(LLMs\) are often used to perform a form of conditional inference\. In\-context learning is one implicit way to do this\. The prompt specifies a context and the model’s output is treated as an estimate of the corresponding conditional distribution\. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities\. In particular, the law of total probability asserts that prior\-weighted conditional distributions aggregate into population\-level marginals over any valid partition of the population\. In this work, we investigate to what extent LLM estimates adhere to this self\-consistency principle\. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine\-grained subpopulations\. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population\-level estimates, and compare them across partitions of varying granularity\. Translating these comparisons into a benchmark score and evaluating frontier models across problem domains, we demonstrate widespread violations of basic consistency properties\. An in\-depth study of persona prompting further reveals a pattern we call the*macro fallacy*: estimates reconstructed from more fine\-grained subpopulation responses are often better aligned with human reference data than direct population\-level estimates\. The effect persists across variations in partitioning, verbalization, and estimation task, suggesting that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates\. This gap establishes*statistical self\-consistency*as an unsaturated, reference\-free criterion for evaluating LLMs\.
## 1Introduction
“Given that a voter lives in California, how likely are they to vote Republican in the next presidential election?” When we pose such a question to a large language model, we typically interpret its answer as an estimate of a conditional probability: the prompt identifies individuals within a subpopulation, and the model estimates the probability of an outcome within this group\. The same interpretation applies across forecasting, social simulation, and other application domains, where contextual information is used to elicit predictions for particular scenarios\. This view also underpins theoretical accounts of in\-context learning \(ICL\), which explain how models can use the information supplied in context to perform conditional inference\([Xie et al\., 2022](https://arxiv.org/html/2607.15277#bib.bib14);[Müller et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib12);[Reuter et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib18)\)\.
Persona prompting offers one use case of in\-context learning where the conditional\-inference interpretation is particularly widespread\. Here, a prompt is augmented with attributes of a person, such as age, occupation, or location, and the model is asked to respond from this perspective\([Argyle et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib2)\)\. Outputs are then expected to approximate a conditional distribution over responses of individuals with a given attribute profile\. As a testament to this perspective, distributional alignment has become a popular evaluation metric\([Santurkar et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib4);[Meister et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib7)\)\.
Elicit LLMestimates via ICLPartition populationinto subgroupsReconstruct aggregates vialaw of total probabilityEvaluate self\-consistencyand alignmentQuestion:What is the probability that their yearly income is above $ 50k?Context:A personat least 20 years old\.Context:A personyounger than 20 years\.Rootestimate=?\\stackrel\{\{\\scriptstyle?\}\}\{\{=\}\}Groundtruth=?\\stackrel\{\{\\scriptstyle?\}\}\{\{=\}\}
Figure 1:Reconstructing aggregate estimates from subpopulation estimates\.A binary tree recursively partitions a base population into increasingly specific subpopulations\. For each subpopulation, we elicit an LLM estimate by verbalizing the split criteria in context, then reconstruct population\-level aggregate estimates at each level via the law of total probability\. We compare the reconstructed aggregates against the model’s direct marginal estimate to assess*self\-consistency*, and against ground\-truth population data to assess*alignment*\.This conditional\-inference interpretation implies that model outputs should obey basic laws of probability theory, yet it remains poorly understood to what extent they do\. As[Cruz et al\. \(2024\)](https://arxiv.org/html/2607.15277#bib.bib16)noted, if language\-model predictions formed a coherent joint distribution, prompting with fewer attributes would yield the same prediction as marginalizing over the omitted attributes\. We evaluate and operationalize this consistency criterion using the law of total probability\. The marginal over any partition must equal the prior\-weighted aggregate of its conditionals\. Persona attributes serve to induce such partitions at different levels of granularity\. Similarly, any pair of complementary events defines a binary partition, such as*inflation rising*versus*not rising\.*Comparing the direct estimate with the prior\-weighted aggregate of the estimates for the partition elements yields a natural, reference\-free self\-consistency check\. While self\-consistency does not require any ground\-truth reference data, persona prompting serves as an appealing testbed because it additionally supplies human reference statistics for alignment analyses and ablations\. We use this framework to stress\-test the often implicit assumption that in\-context learning performs conditional inference\.
### 1\.1Our work
We start with an in\-depth study of persona prompting as an instance of in\-context learning\. We use*binary conditioning trees*to systematically construct personas that define subpopulations at increasing levels of specificity \(see[Figure1](https://arxiv.org/html/2607.15277#S1.F1)\)\. Let𝒳\\mathcal\{X\}denote a base population\. Each successive level of the tree refines the persona description with an additional attribute\. By construction, the nodes at that level define a partition𝒫\\mathcal\{P\}of the population, with𝒳=∪𝒮∈𝒫𝒮\\mathcal\{X\}=\\cup\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\}\\,\\mathcal\{S\}\. Given such a partition, for any eventEE, the*law of total probability*\([Bertsekas and Tsitsiklis, 2008](https://arxiv.org/html/2607.15277#bib.bib23), cf\.\)asserts that
ℙ\(E\|X∈𝒳\)⏟aggregate=∑𝒮∈𝒫γ𝒮⏟priorℙ\(E\|X∈𝒮\)⏟conditional,\\underbrace\{\\mathbb\{P\}\(E\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{X\}\)\}\_\{\\text\{aggregate\}\}=\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\}\\underbrace\{\\gamma\_\{\\mathcal\{S\}\}\}\_\{\\text\{prior\}\}\\;\\underbrace\{\\mathbb\{P\}\(E\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)\}\_\{\\text\{conditional\}\},\(2\)whereγ𝒮∈\[0,1\]\\gamma\_\{\\mathcal\{S\}\}\\in\[0,1\]denotes the relative size of partition element𝒮\\mathcal\{S\}\. This equivalence is the core observation we build on, and we apply it in two ways:
1. 1\.We substitute LLM estimates into the right\-hand side of equation \([2](https://arxiv.org/html/2607.15277#S1.E2)\) and compare the resulting aggregate with reference data\. We use in\-context learning to elicit the conditional probabilities and combine them using LLM\-estimated prior weights\. Comparing the resulting aggregate estimates for different partitions against ground\-truth data reveals an interesting pattern, which we call the*macro fallacy*\. Namely, the aggregateℙ\(E\|X∈𝒳\)\\mathbb\{P\}\(E\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{X\}\)retrieved directly from the LLM is systematically less aligned with human data than the estimate reconstructed from finer\-grained conditionals and the corresponding priors via the law of total probability\. The effect is robust across target quantities, models, and tree structures\.
2. 2\.We compare LLM estimates of the right\-hand side of \([2](https://arxiv.org/html/2607.15277#S1.E2)\) across different partitions\. Because the law of total probability requires these quantities to be equal for every partition, any disagreement reveals an internal inconsistency in the model\. We propose two complementary diagnostics of statistical self\-consistency:*split consistency*and*order consistency*\. Together, these checks assess statistical self\-consistency across a broad class of aggregation procedures, without requiring any ground\-truth data\. Instantiating our consistency checks in settings spanning persona prompting and synthetic forecasting across a diverse set of models, we find widespread violations even among state\-of\-the\-art frontier models\. Furthermore, consistency scores exhibit no apparent correlation with the abilities assessed by standard benchmarks\.
## 2Background and related work
#### In\-context learning as conditional inference\.
We use*in\-context learning*\(ICL\) in the broad sense, which encompasses general forms of learning from contextual information\. This information may consist of natural language instructions, task demonstrations, or both, and includes common settings such as zero\-shot prompting, few\-shot prompting, persona prompting, and forecasting\([Brown et al\., 2020](https://arxiv.org/html/2607.15277#bib.bib11);[Schulhoff et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib37)\)\. A substantial body of literature interprets ICL as conditional or Bayesian inference\([Xie et al\., 2022](https://arxiv.org/html/2607.15277#bib.bib14);[Müller et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib12);[Reuter et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib18);[de Carvalho et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib13);[Buening et al\., 2026](https://arxiv.org/html/2607.15277#bib.bib28)\)\. However, there is mixed evidence on how closely LLMs satisfy this characterization\([Falck et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib19);[Chlon et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib15)\)\.
#### Persona conditioning and distributional alignment\.
Persona conditioning is a form of ICL in which a model is asked to respond from the perspective of a person described by sociodemographic attributes\([Argyle et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib2);[Beck et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib5)\)\. Distributional alignment studies compare the resulting response distributions with those of human populations, thereby implicitly treating model outputs as conditional distributions\([Santurkar et al\., 2023](https://arxiv.org/html/2607.15277#bib.bib4);[Meister et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib7);[Ma et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib40);[Hu et al\., 2026](https://arxiv.org/html/2607.15277#bib.bib21)\)\. This interpretation has been challenged by unreliable simulations, sensitivity to prompt formulation, degenerate output distributions, and distortions of the joint structure of human beliefs\([Dominguez\-Olmedo et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib8);[Bisbee et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib3);[Cruz et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib16);[Lutz et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib6);[Barrie and Cerina, 2026](https://arxiv.org/html/2607.15277#bib.bib17)\)\. We offer a new perspective on this discussion by asking whether statistically equivalent estimators derived from the same model agree across levels of granularity\.
#### Statistical self\-consistency\.
We define statistical self\-consistency as agreement between a marginal estimate and the prior\-weighted aggregate of its conditionals over any valid partition, a requirement that does not rely on reference data\. This differs from the prevalent uses of the term self\-consistency in the machine learning literature, where it typically refers to consistency across semantically equivalent contexts\([Elazar et al\., 2021](https://arxiv.org/html/2607.15277#bib.bib31);[Wang et al\., 2022](https://arxiv.org/html/2607.15277#bib.bib10);[Li et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib32)\)\. More closely related work tests probabilistic coherence under conjunctions, Bayes’ rule, or logical forecasting constraints\([Zhu and Griffiths, 2024](https://arxiv.org/html/2607.15277#bib.bib1);[Paruchuri et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib9);[Fluri et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib30);[Paleka et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib29)\), as well as agreement between direct and compositionally reconstructed answers\([Bottman et al\., 2026](https://arxiv.org/html/2607.15277#bib.bib36)\)\. What is unique to our work is that we compare estimates across levels of specificity to assess coherence and use ground\-truth data to examine the structure of consistency violations\.
## 3Expressing aggregates through partitions
To instantiate the law of total probability, we systematically slice a population into recursively defined and mutually exclusive subpopulations, and then assess the consistency of LLM estimates aggregated over the partitions induced at different tree levels\.
### 3\.1Constructing partitions using binary trees
We use a binary conditioning tree \(BCT\) to recursively partition a base population into increasingly fine\-grained subpopulations\.111We refer to a population in a statistical sense as an arbitrary distribution over some instance space𝒳\\mathcal\{X\}\.The root node corresponds to the base population𝒳\\mathcal\{X\}, and each levelℓ\\ellapplies the same binary split functionϕℓ:𝒳→\{0,1\}\\phi\_\{\\ell\}:\\mathcal\{X\}\\rightarrow\\\{0,1\\\}to every node from the preceding level\. Data points that satisfyϕℓ\(x\)=0\\phi\_\{\\ell\}\(x\)=0are assigned to the left child, while data points withϕℓ\(x\)=1\\phi\_\{\\ell\}\(x\)=1are assigned to the right child\. As a result, the tree induces a different partition of𝒳\\mathcal\{X\}at every level\.
#### Persona trees as a testbed\.
As a case study, we construct a binary tree over the population of individuals living in the United States, using data from the 2024 American Community Survey \(ACS\)222[https://www\.census\.gov/programs\-surveys/acs](https://www.census.gov/programs-surveys/acs)as the human reference population\. The ACS dataset contains records for 3\.8 million individuals and captures demographic, employment, health, and related attributes\. Due to its large sample size, rigorous sampling design, and population weighting, it offers one of the best representations of the U\.S\. population\. For a fixed set of attributes, we select balanced splits that reduce the prior\-weighted within\-subgroup variance of yearly income, analogous to regression trees\([Breiman et al\., 1984](https://arxiv.org/html/2607.15277#bib.bib27)\)\. This yields a split by ageϕ1\(x\)=𝟙\{𝟛𝟙≤age\(𝕩\)≤𝟞𝟠\}\\phi\_\{1\}\(x\)=\\mathbbold\{1\}\\\{31\\leq\\operatorname\{age\}\(x\)\\leq 68\\\}, followed by employment statusϕ2\(x\)=𝟙\{𝕩is employed or self\-employed\}\\phi\_\{2\}\(x\)=\\mathbbold\{1\}\\\{x\\text\{ is employed or self\-employed\}\\\}\. In[SectionF\.1](https://arxiv.org/html/2607.15277#A6.SS1), we provide additional details on split selection and data preprocessing, including the ACS attribute mappings and income binning\.
### 3\.2Obtaining conditional LLM estimates at varying levels of granularity
For our ACS case study, we instantiate the event of interestEEas the income exceeding a specified threshold, which mimics the popular UCI Adult benchmark task\([Becker and Kohavi, 1996](https://arxiv.org/html/2607.15277#bib.bib38);[Ding et al\., 2021](https://arxiv.org/html/2607.15277#bib.bib22)\)\. Specifically, for a thresholdτ∈ℝ\\tau\\in\\mathbb\{R\}, we takeEτ=\{Y\>τ\}E\_\{\\tau\}=\\\{Y\>\\tau\\\}, whereYYdenotes yearly income\. For any subpopulation𝒮⊆𝒳\\mathcal\{S\}\\subseteq\\mathcal\{X\}, we define the corresponding tail probability as
Tτ\(𝒮\)≜ℙ\(Y\>τ\|X∈𝒮\)\.T\_\{\\tau\}\(\\mathcal\{S\}\)\\triangleq\\mathbb\{P\}\(Y\>\\tau\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)\.\(3\)We denote the LLM estimate of this quantity byT^τ\(𝒮\)\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)\. Varyingτ\\taulets us probe binary target distributions with different degrees of entropy\. The entropy is maximized whenTτ\(𝒮\)=1/2T\_\{\\tau\}\(\\mathcal\{S\}\)=1/2and decreases as the tail probability approaches zero or one\.
#### Verbalizing split criteria for in\-context learning\.
To implement this conditioning structure, we pair the estimation query with a natural\-language description of the subpopulation specific to each BCT node\. This description consists of a base contextC0C\_\{0\}specifying the reference population, followed by the verbalized branch restrictions along the root\-to\-node path\. This construction follows common practice in sociodemographic prompting\([Beck et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib5)\)\. When the context consists only ofC0C\_\{0\}, we refer to the resulting method as*direct prompting*\. The base contextC0C\_\{0\}may also be empty, in which case the base population is left implicit\. For the persona tree, we use the folktexts package\([Cruz et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib16)\)to systematically translate attributes into persona descriptions\. The exact prompt template is given in[SectionH\.2](https://arxiv.org/html/2607.15277#A8.SS2), with prompt\-sensitivity ablations reported in[SectionsE\.5](https://arxiv.org/html/2607.15277#A5.SS5)and[E\.6](https://arxiv.org/html/2607.15277#A5.SS6)\.
### 3\.3Reconstructing aggregate estimates
Aggregating conditional estimates according to the law of total probability also requires subgroup priors, which we elicit from LLMs using a level\-wise protocol\. For each tree level, we query the model for the prior distribution over the corresponding partition, repeat this multiple times, renormalize the raw outputs, and average the resulting estimates\. We denote the resulting LLM\-estimated prior of subpopulation𝒮\\mathcal\{S\}byγ^𝒮\\hat\{\\gamma\}\_\{\\mathcal\{S\}\}\. Further details are provided in[SectionF\.2](https://arxiv.org/html/2607.15277#A6.SS2)\. Combining the elicited conditionals and priors yields an LLM\-reconstructed aggregate at each levelℓ\\ell:
T^τ,agg\(ℓ\)\(𝒳\)≜∑𝒮∈𝒫ℓ\(𝒳\)γ^𝒮T^τ\(𝒮\)\.\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}^\{\(\\ell\)\}\(\\mathcal\{X\}\)\\triangleq\{\\sum\}\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\hat\{\\gamma\}\_\{\\mathcal\{S\}\}\\,\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)\.\(4\)Under the conditional\-inference interpretation of in\-context learning, each node prompt queries the same target outcome for a specific subpopulation\. The branch restrictions along the root\-to\-node path act as conditioning events that define this subpopulation relative to the base population\. Under these assumptions, the elicited quantitiesT^τ\(𝒮\)\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)andγ^𝒮\\hat\{\\gamma\}\_\{\\mathcal\{S\}\}should satisfy the same aggregation identity as their survey\-based counterparts\.
## 4The macro fallacy
0123456Level−\-50%0%50%Aggregation gain0123456Level−\-50%0%50%Aggregation gainGPT\-5\.43\.1 Pro3 FlashOpus 4\.74\.203\.6 Plusτ=100\\tau=100USDτ=10k\\tau=10\\mathrm\{k\}USDτ=20k\\tau=20\\mathrm\{k\}USDτ=40k\\tau=40\\mathrm\{k\}USDτ=60k\\tau=60\\mathrm\{k\}USDτ=80k\\tau=80\\mathrm\{k\}USD
\(a\)Thresholdτ=40k\\tau=40\\mathrm\{k\}USD for varying models\.\(b\)GPT\-5\.4 with varying income threshold\.
Figure 2:Relative error reduction of reconstructed aggregates over direct prompting\.We report the relative aggregation gain of the reconstructed estimate, defined as1−AErr\(ℓ\)/AErr\(0\)1\-\\mathrm\{AErr\}\(\\ell\)/\\mathrm\{AErr\}\(0\), as a function of tree depthℓ\\ell, whereAErr\(0\)\\mathrm\{AErr\}\(0\)denotes the alignment error of direct prompting\. The black error bars indicate 90 % confidence intervals computed by bootstrapping with 1000 samples\. The orange curve shows the average aggregation gain\. Reconstructing aggregates generally improves alignment in the first few tree levels, but the gain diminishes with depth and can become negative at deeper tree levels\.Each level of the persona tree induces a different partition of the base population and comes with a corresponding LLM\-reconstructed aggregate estimate, computed according to \([4](https://arxiv.org/html/2607.15277#S3.E4)\)\. We start by comparing each of these estimates against survey\-based reference data to understand how well they are aligned with ground\-truth population statistics\. More formally, for a population𝒳\\mathcal\{X\}and the partition𝒫ℓ\(𝒳\)\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)induced by levelℓ\\ell, we define the aggregate alignment error between the reconstructed population\-level estimateT^τ,agg\(ℓ\)\(𝒳\)\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}^\{\(\\ell\)\}\(\\mathcal\{X\}\)and the ACS ground\-truth tail probabilityTτ\(𝒳\)T\_\{\\tau\}\(\\mathcal\{X\}\)as
AErr\(ℓ\)≜\|Tτ\(𝒳\)−T^τ,agg\(ℓ\)\(𝒳\)\|\.\\text\{AErr\}\(\\ell\)\\triangleq\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\\kern 1\.00006ptT\_\{\\tau\}\(\\mathcal\{X\}\)\-\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}^\{\(\\ell\)\}\(\\mathcal\{X\}\)\\kern 1\.00006pt\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\.\(5\)At levelℓ=0\\ell=0, the definition corresponds to a classical notion of alignment between the model and the data\([Meister et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib7)\)\. At deeper levels, the LLM estimate is instead obtained by first eliciting subgroup\-level conditionals and then aggregating them back to the population level by using prior estimates\. Thus,AErr\(ℓ\)\\text\{AErr\}\(\\ell\)measures how well the same target quantity is recovered when the prompt is posed at the granularity induced by tree levelℓ\\ell\. For estimates adhering to probability axioms, the error should not depend onℓ\\ell\. However, we find that invariance fails in practice and the alignment error varies substantially across levels\.
[Figure2](https://arxiv.org/html/2607.15277#S4.F2)shows the reduction in alignment error at deeper levels relative to direct prompting across models and problems\. We refer to this reduction as the aggregation gain and use it to highlight a pattern we call the*macro fallacy*\. Namely, direct root\-level estimates often exhibit larger alignment errors relative to the human baseline than reconstructed estimates at lower levels\. In our experiment, this pattern holds on average up to level four across models \([Figure2\(a\)](https://arxiv.org/html/2607.15277#S4.F2.sf1)\) and target distributions of varying entropy \([Figure2\(b\)](https://arxiv.org/html/2607.15277#S4.F2.sf2)\)\. We further observe that the aggregation gain profile appears concave, with a peak between levels one and four\. Remarkably, this improvement arises solely from decomposing and recombining the population estimates without injecting any ground\-truth statistics\. Additional results, ablations, and experimental details are presented in[AppendicesB](https://arxiv.org/html/2607.15277#A2),[G\.1](https://arxiv.org/html/2607.15277#A7.SS1),[F\.3](https://arxiv.org/html/2607.15277#A6.SS3)and[E\.6](https://arxiv.org/html/2607.15277#A5.SS6)\.
### 4\.1Disentangling error sources in aggregate estimates
To better understand the findings of the previous section, we examine the quality of the conditionals and the priors separately\.[Figure3\(a\)](https://arxiv.org/html/2607.15277#S4.F3.sf1)shows that conditional estimates tend to improve with depth\. The improvement holds for all nodes at the first two levels, and less consistently at level three\. However, the two nodes that exhibit large alignment errors have negligible prior mass and contribute little to the prior\-weighted average\. Conversely, priors become harder to estimate at greater depths, as reflected in the increasing total variation distance between estimated and survey\-based priors across models \([Figure3\(b\)](https://arxiv.org/html/2607.15277#S4.F3.sf2)\)\. Together, these patterns suggest that improved conditional estimates drive aggregation gains, while errors in population weighting increasingly offset the benefits of finer conditioning\.
33%0%10%9%7%0%12%29%000\.250\.250\.500\.50
\(a\)Node\-wise conditional alignment error\. In lower levels, nodes with high alignment error tend to have small priors\.
1234Level0\.000\.000\.250\.250\.500\.50TV distance5\.44o mini3\.1 ProSonnet 4\.6
\(b\)Distance between LLM\-estimated priors and ground\-truth values\. Priors get less accurate with depth\.
11223344Level4\.04\.04\.54\.55\.05\.0Within\-group var\.×109\\times 10^\{9\}0\.00\.00\.50\.51\.01\.0Cross\-group var\.×109\\times 10^\{9\}
\(c\)Average within\-subgroup variance decreases with depth \(green\) while cross\-subgroup variance \(orange\) increases\.
Figure 3:Tracing the origin of the macro fallacy\.Panels[3\(a\)](https://arxiv.org/html/2607.15277#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2607.15277#S4.F3.sf2)separate the subgroup conditional estimates and subgroup priors in the reconstructed aggregate estimate\. Node\-wise conditional errors tend to improve over the root node, yet they vary substantially across the tree\. Panel[3\(c\)](https://arxiv.org/html/2607.15277#S4.F3.sf3)illustrates the variance decomposition\. As the population is refined, residual heterogeneity within conditioned subgroups decreases, while systematic differences between subgroups become more explicit\.Why do conditional estimates improve with depth? One possible explanation is that partitioning reduces the burden of expressing heterogeneity in a single estimate, which is something LLMs struggle with\([Bisbee et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib3)\)\. Partitioning the population mitigates this issue by reducing average residual heterogeneity within subgroups while making systematic differences between them explicit\. The law of total variance formalizes this decomposition as𝕍\(Y\)=𝔼\[𝕍\(Y\|A\)\]\+𝕍\(𝔼\[Y\|A\]\)\\mathbb\{V\}\(Y\)=\\mathbb\{E\}\[\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\)\]\+\\mathbb\{V\}\(\\mathbb\{E\}\[Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\]\), whereAAidentifies subgroup membership\. The within\-group variance,𝔼\[𝕍\(Y\|A\)\]\\mathbb\{E\}\[\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\)\], is the average variance within subgroups, while the cross\-group variance,𝕍\(𝔼\[Y\|A\]\)\\mathbb\{V\}\(\\mathbb\{E\}\[Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\]\), captures variation in subgroup means\. As the partition is refined, variation shifts from the within\-group component to the cross\-group component, as illustrated in[Figure3\(c\)](https://arxiv.org/html/2607.15277#S4.F3.sf3)\. Explicit decomposition thus reduces the burden of representing heterogeneous populations while directing the model’s attention to individual subpopulations\.
## 5Statistical self\-consistency
A consistent conditional estimator should satisfy the aggregation identities implied by the law of total probability over any valid partition, across trees and problem domains\. In this section, we turn these criteria into two finite collections of self\-consistency checks\.*Split consistency*tests whether estimates elicited at two adjacent levels of a conditioning tree aggregate according to the law of total probability\.*Order consistency*tests whether direct estimates are invariant under permutations of the order in which the conditioning constraints are verbalized\.
#### General target functionals\.
For a feature vectorX∈𝒳X\\in\\mathcal\{X\}, a subpopulation𝒮⊆𝒳\\mathcal\{S\}\\subseteq\\mathcal\{X\}, and a measurable functionhhof the outcomeYY, we define the target functional as
Th\(𝒮\)≜𝔼\[h\(Y\)\|X∈𝒮\]\.T\_\{h\}\(\\mathcal\{S\}\)\\triangleq\\mathbb\{E\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\[$\}\}\}h\(Y\)\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\]$\}\}\}\.\(6\)Tail probabilities are recovered as the special caseh:y↦𝟙\{𝕪\>τ\}h:y\\mapsto\\mathbbold\{1\}\\\{y\>\\tau\\\}\. Crucially, this functional obeys the aggregation identity induced by the law of total probability and is invariant to the ordering of the constraints defining𝒮\\mathcal\{S\}\. We useT^h\(𝒮\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)to refer to the LLM estimate elicited under a fixed canonical ordering and we useT^h\(𝒮,π\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi\)whenever an alternative orderingπ\\piis applied to verbalize the constraints\.
### 5\.1Split consistency
Split consistency asks whether an estimate is preserved when reconstructed over a binary attribute split\. Given a subpopulation𝒮⊆𝒳\\mathcal\{S\}\\subseteq\\mathcal\{X\}and a binary attributeAA, we introduce the notationT^h,agg\(𝒮,A\)\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\};A\)for the prior\-weighted aggregate over the two subpopulations𝒮0≜𝒮∩\{A=0\}\\mathcal\{S\}\_\{0\}\\triangleq\\mathcal\{S\}\\cap\\\{A=0\\\}and𝒮1≜𝒮∩\{A=1\}\\mathcal\{S\}\_\{1\}\\triangleq\\mathcal\{S\}\\cap\\\{A=1\\\}\. Split consistency quantifies the discrepancy between this aggregate and the direct estimate at𝒮\\mathcal\{S\}using a metricdist\(∙,∙\)\\dist\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt,\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\)on the space of elicited estimates\.
###### Definition 1\(Split consistency\)\.
Let𝒮⊆𝒳\\mathcal\{S\}\\subseteq\\mathcal\{X\}be a subpopulation andAAa binary attribute\. The pair\(𝒮,A\)\(\\mathcal\{S\},A\)isε\\varepsilon\-split consistent if
dist\(T^h\(𝒮\),T^h,agg\(𝒮,A\)\)<ε\.\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\};A\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\varepsilon\.\(7\)
Beyond certifying a single attribute split, we are interested in an aggregate score over a broader set of possible splits occurring in a binary tree induced by a set of multiple attributes𝒜\\mathcal\{A\}\. We thus collect all pairs of conditioning events and admissible next splits induced by𝒜\\mathcal\{A\}and summarize them into a consistency score\. We use𝒮\(𝒥,𝒂𝒥\)≜⋂j∈𝒥\{Aj=aj\}\\mathcal\{S\}\(\\mathcal\{J\},\\bm\{a\}\_\{\\mathcal\{J\}\}\)\\triangleq\{\\bigcap\}\_\{j\\in\\mathcal\{J\}\}\\\{A\_\{j\}=a\_\{j\}\\\}as a shorthand for the conditioning event defined by a given realization of attributes within the index set𝒥\\mathcal\{J\}\.
###### Definition 2\(Split consistency score\)\.
Given a set of attributes𝒜=\{A1,…,Ad\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{d\}\\\}and a thresholdε\\varepsilon, the split consistency score is the fraction of satisfied checks
SCε\(𝒜\)≜1\|𝒞SC\(𝒜\)\|∑\(𝒮,Ak\)∈𝒞SC\(𝒜\)𝟙\{dist\(𝕋^𝕙\(𝒮\),𝕋^𝕙,agg\(𝒮;𝔸𝕜\)\)<ε\},\\textnormal\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)\\triangleq\\frac\{1\}\{\|\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\)\|\}\\sum\_\{\(\\mathcal\{S\},A\_\{k\}\)\\in\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\)\}\\mathbbold\{1\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\{$\}\}\}\\\!\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\};A\_\{k\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\varepsilon\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\}$\}\}\},\(8\)with𝒞SC\(𝒜\)≜\{\(𝒮\(𝒥,𝐚𝒥\),Ak\):𝒥⊆\[d\],𝐚𝒥∈\{0,1\}\|𝒥\|,k∉𝒥\}\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\)\\triangleq\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\{$\}\}\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\mathcal\{S\}\(\\mathcal\{J\},\\bm\{a\}\_\{\\mathcal\{J\}\}\),A\_\{k\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}:\\mathcal\{J\}\\subseteq\[d\],\\;\\bm\{a\}\_\{\\mathcal\{J\}\}\\in\\\{0,1\\\}^\{\|\\mathcal\{J\}\|\},\\;k\\notin\\mathcal\{J\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\}$\}\}\}\.
The split consistency score counts the fraction of split consistency checks passed by the model\.SCε\(𝒜\)=1\\textnormal\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)=1corresponds to the requirement that every admissible split consistency check is satisfied, while intermediate values quantify partial consistency\. This makes the criterion useful as a benchmark score even when no model satisfies all checks\.
### 5\.2Order consistency
Split consistency alone does not make an estimate statistically self\-consistent\. A complementary requirement is that the estimate assigned to a conditioning event should not depend on the order in which its defining constraints are applied\. Such invariance cannot be taken for granted, as LLMs are sensitive to reordered but semantically equivalent inputs\([Chen et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib34);[Guan et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib35)\)\. We formalize order invariance as a self\-consistency check over a set of binary attributes\.
###### Definition 3\(Order consistency\)\.
Let𝒮=𝒮\(𝒥,𝐚𝒥\)\\mathcal\{S\}=\\mathcal\{S\}\(\\mathcal\{J\},\\bm\{a\}\_\{\\mathcal\{J\}\}\)be a conditioning event with\|𝒥\|≥2\|\\mathcal\{J\}\|\\geq 2\. LetΠ\(𝒥\)\\Pi\(\\mathcal\{J\}\)denote the set of orderings of its defining constraints\. We say that𝒮\\mathcal\{S\}isε\\varepsilon\-order consistent if
dist\(T^h\(𝒮,π\),T^h\(𝒮,π′\)\)<ε\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi\),\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\varepsilon\(9\)for every pair of distinct orderingsπ,π′∈Π\(𝒥\)\\pi,\\pi^\{\\prime\}\\in\\Pi\(\\mathcal\{J\}\)\.
Checking[Definition3](https://arxiv.org/html/2607.15277#Thmtheorem3)across all\|𝒥\|\!\\lvert\\mathcal\{J\}\\rvert\!constraint orderings quickly becomes infeasible with increasing depth of the binary conditioning tree\. We therefore restrict the evaluation to conditioning events involving exactly two attributes and construct an order consistency score defined as the fraction of satisfied checks\.
###### Definition 4\(Order consistency score\)\.
Given an attribute set𝒜=\{A1,…,Ad\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{d\}\\\}, the order consistency score is the fraction of checks satisfied at toleranceε\\varepsilon,
OCε\(𝒜\)≜1\|𝒞OC\(𝒜\)\|∑𝒮∈𝒞OC\(𝒜\)\{dist\(𝕋^𝕙\(𝒮;π𝟙\(𝒮\)\),𝕋^𝕙\(𝒮;π𝟚\(𝒮\)\)\)<ε\},\\textnormal\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)\\triangleq\\frac\{1\}\{\\left\|\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)\\right\|\}\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)\}\\mathbbold\{1\}\\\!\\left\\\{\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\mathcal\{S\};\\pi\_\{1\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\},\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\mathcal\{S\};\\pi\_\{2\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\varepsilon\\right\\\},\(10\)where𝒞OC\(𝒜\)≜\{𝒮\(\{i,j\},\(ai,aj\)\):1≤i<j≤d,\(ai,aj\)∈\{0,1\}2\}\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)\\triangleq\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\{$\}\}\}\\mathcal\{S\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\\{i,j\\\},\(a\_\{i\},a\_\{j\}\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}:1\\leq i<j\\leq d,\\;\(a\_\{i\},a\_\{j\}\)\\in\\\{0,1\\\}^\{2\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\\\}$\}\}\}andπ1\(𝒮\)\\pi\_\{1\}\(\\mathcal\{S\}\)andπ2\(𝒮\)\\pi\_\{2\}\(\\mathcal\{S\}\)denote the two possible constraint orderings of the two\-attribute event𝒮\\mathcal\{S\}\.
Restricting the score to pairwise checks makes the evaluation tractable\. In the next subsection, we state the condition under which these checks extend to arbitrary conditioning events\.
### 5\.3Consistency of reconstructed aggregates
We now lift the local self\-consistency checks to a uniform bound on the discrepancy between aggregates computed at different levels of arbitrary BCTs\. This result needs two assumptions about the elicitation protocol and the model, which we formalize below\.
###### Assumption 1\(Shared aggregation weights\)\.
For each partition, prior weights are elicited once using a fixed canonical ordering of the attributes and reused for every aggregation over that partition, independently of the ordering used to elicit the conditional estimates\.
###### Assumption 2\(Context\-independent pairwise order discrepancy\)\.
For any pair of attributesAi,AjA\_\{i\},A\_\{j\}, any assignmentsai,aja\_\{i\},a\_\{j\}, and any \(possibly empty\) prefix, intervening, and suffix conditioning contextsPP,QQ, andRR, the discrepancy
dist\(T^h\(P,Ai=ai,Q,Aj=aj,R\),T^h\(P,Aj=aj,Q,Ai=ai,R\)\)\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(P,A\_\{i\}=a\_\{i\},Q,A\_\{j\}=a\_\{j\},R\),\\hat\{T\}\_\{h\}\(P,A\_\{j\}=a\_\{j\},Q,A\_\{i\}=a\_\{i\},R\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\(11\)depends only on the constraintsAi=aiA\_\{i\}=a\_\{i\}andAj=ajA\_\{j\}=a\_\{j\}, and is therefore independent of the surrounding contextsPP,QQ, andRR\.
[Proposition5](https://arxiv.org/html/2607.15277#Thmtheorem5)explains how the two proposed checks complement each other under the assumption of robust prior elicitation and context independence of pairwise order consistency\. More details and the proof are given in[SectionD\.3](https://arxiv.org/html/2607.15277#A4.SS3)\.
###### Proposition 5\(Informal uniform consistency bound across all trees and levels\)\.
Suppose that[Assumptions2](https://arxiv.org/html/2607.15277#Thmassumption2)and[1](https://arxiv.org/html/2607.15277#Thmassumption1)hold and that the distance metric is jointly convex under mixtures\.333A metricdist\(∙,∙\)\\dist\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt,\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\)is jointly convex under mixtures if, for any weightsλ1,…,λn≥0\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\\geq 0satisfying∑i=1nλi=1\\sum\_\{i=1\}^\{n\}\\lambda\_\{i\}=1, it holds thatdist\(∑iλiPi,∑iλiQi\)≤∑iλidist\(Pi,Qi\)\.\\dist\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\{\\sum\}\_\{i\}\\lambda\_\{i\}P\_\{i\},\{\\sum\}\_\{i\}\\lambda\_\{i\}Q\_\{i\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}\\leq\\sum\_\{i\}\\lambda\_\{i\}\\,\\dist\(P\_\{i\},Q\_\{i\}\)\.Assume that every split consistency check in𝒞SC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\)is satisfied at toleranceεSC\\varepsilon\_\{\\textnormal\{SC\}\}, every pairwise order consistency check in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)is satisfied at toleranceεOC\\varepsilon\_\{\\textnormal\{OC\}\}, and every elicited prior at one level deviates by at mostεPR\\varepsilon\_\{\\textnormal\{PR\}\}from the prior implied by marginalizing the elicited prior at the next level\. Letπ\\piandπ′\\pi^\{\\prime\}be any two constraint orderings that define BCTs induced by𝒜\\mathcal\{A\}, both rooted at the same base population𝒳\\mathcal\{X\}\. If both trees have depth at mostdd, their corresponding level aggregates satisfy
dist\(T^h,agg,𝒯\(ℓ\)\(𝒳,π\),T^h,agg,𝒯′\(ℓ′\)\(𝒳,π′\)\)<2dεSC\+2\(d−1\)DhεPR\+2\(d−1\)εOC,\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\mathrm\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\mathrm\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\\right\)<2d\\,\\varepsilon\_\{\\textnormal\{SC\}\}\+2\(d\-1\)D\_\{h\}\\,\\varepsilon\_\{\\textnormal\{PR\}\}\+2\(d\-1\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{OC\}\},\(12\)for every admissible pair of levelsℓ,ℓ′\\ell,\\ell^\{\\prime\}\. Here,DhD\_\{h\}is the finite diameter of the admissible estimate space, withDh=1D\_\{h\}=1for normalized distance metrics\.
The bound separates three sources of discrepancy: errors in split consistency, errors in prior elicitation across levels, and errors in pairwise order consistency\. Under the stated assumptions, controlling these errors certifies that level aggregates remain close across different constraint orderings and levels\.
GPT\-4o miniGPT\-4oGPT\-5 miniGPT\-5\.4 nanoGPT\-4\.1GPT\-5\.1GPT\-5\.2GPT\-5\.6 LunaGPT\-5\.4GPT\-5\.5GPT\-5\.6 SolQwen 2\.5 7B Instr\.Qwen 3 8BQwen 3\.5 9BQwen 3\.5 35BQwen 3\.5 122BQwen 3\.5 27BQwen 3\.7 PlusQwen 3\.6 PlusQwen 3\.7 MaxGemini 3 FlashGemini 3\.1 ProSonnet 4\.6Opus 4\.70\.00\.00\.20\.20\.40\.40\.60\.60\.80\.81\.01\.0Consistency scoreSplit consistencyOrder consistencyArtificial Analysis Intelligence Index0010102020303040405050AAI Index
Figure 4:Self\-consistency plateaus across model generations\.We report the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02for the thresholded ACS income prediction task withτ=40k\\tau=40\\mathrm\{k\}USD\. The gray shaded area shows the Artificial Analysis Intelligence \(AAI\) Index\([Artificial Analysis, 2026](https://arxiv.org/html/2607.15277#bib.bib33)\)as a proxy for general model capability\. Neither split nor order consistency improves systematically with increasing model capability\.
## 6Evaluation
Next, we demonstrate the versatility of our self\-consistency checks and evaluate them across different tasks\. Our self\-consistency checks have the appealing property that they are reference\-free, which allows us to extend our evaluation beyond surveys to domains where reliable labels are expensive, unavailable, or ill\-defined\. We instantiatedist\(∙,∙\)\\dist\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt,\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\)as the absolute difference for binary targets and the normalized Wasserstein\-11distance for ordered categorical targets \([SectionC\.1](https://arxiv.org/html/2607.15277#A3.SS1)\)\.
#### Self\-consistency in persona prompting\.
First, we evaluate self\-consistency in the ACS setting discussed in prior sections and examine whether self\-consistency improves with general model capabilities assessed by standard benchmarks\. We take𝒜\\mathcal\{A\}to be the demographic attributes underlying the tree in[Section3](https://arxiv.org/html/2607.15277#S3)\. For every conditioning event induced by𝒜\\mathcal\{A\}, we elicit the model’s estimated tail probability on the thresholded income task withτ=40k\\tau=40\\mathrm\{k\}USD\. We then apply the self\-consistency checks of[Section5](https://arxiv.org/html/2607.15277#S5)to the resulting estimates\.[Figure4](https://arxiv.org/html/2607.15277#S5.F4)shows that, despite large increases in the Artificial Analysis Intelligence \(AAI\) Index\([Artificial Analysis, 2026](https://arxiv.org/html/2607.15277#bib.bib33)\)within model families, neither split nor order consistency improves systematically, with order consistency remaining generally higher\. Statistical self\-consistency therefore does not appear to emerge naturally with stronger overall model performance and remains far from saturated among current models\. Extending this evaluation to decision models in[SectionE\.3](https://arxiv.org/html/2607.15277#A5.SS3)reveals that even models trained specifically for decision tasks violate basic probabilistic consistency requirements\. These findings establish statistical self\-consistency as a distinct dimension of model quality, complementary to the capabilities measured by contemporary benchmarks\. Implementation details and more results are deferred to[SectionsE\.2](https://arxiv.org/html/2607.15277#A5.SS2)and[F\.7](https://arxiv.org/html/2607.15277#A6.SS7)\.
#### Self\-consistency in forecasting problems\.
Recent work on LLM forecasting, such as Prophet Arena\([Yang et al\., 2026](https://arxiv.org/html/2607.15277#bib.bib25)\), studies language models as probabilistic forecasters of real\-world events and evaluates their predictions using proper scoring rules, calibration, and market\-based baselines\. Prior work also tests logical relations such as negation and conjunction, using consistency to diagnose failures when ground truth is unavailable\([Fluri et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib30);[Paleka et al\., 2025](https://arxiv.org/html/2607.15277#bib.bib29)\)\. We ask whether forecasts compose coherently across explicit partitions of the conditioning space\. We study this requirement in two synthetic settings with binary variables encoding contextual information\. In the tennis task, court surface and first\-set outcome refine a prediction for a Federer–Nadal match\. In the fictional fantasy\-combat task, time of day and wind conditions affect the probability of a successful action\. The results in[Tables1\(a\)](https://arxiv.org/html/2607.15277#S6.T1.st1)and[1\(b\)](https://arxiv.org/html/2607.15277#S6.T1.st2)mirror those for the ACS: self\-consistency failures remain widespread across both tasks\. Thus, even intuitive forecasting conditions fail to induce a coherent system of conditional forecasts\. The fantasy\-combat task further highlights the reference\-free nature of self\-consistency\. Although its fictional setting provides no external ground truth against which to assess alignment, its internal consistency can still be evaluated systematically\. Further details appear in[SectionsF\.10](https://arxiv.org/html/2607.15277#A6.SS10)and[F\.11](https://arxiv.org/html/2607.15277#A6.SS11)\.
\(a\)Tennis forecasting
\(b\)Fantasy combat
Table 1:Self\-consistency in forecasting\.For each task, we report the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)forε=0\.02\\varepsilon=0\.02\. Higher values indicate stronger self\-consistency\. Self\-consistency violations are widespread even in simple synthetic settings\.
## 7Conclusion
We investigate how closely the probability estimates obtained from LLMs through in\-context learning conform to the law of total probability\. To do so, we recursively partition populations into increasingly fine\-grained subpopulations and compare two estimates of the same quantity derived at different levels of granularity\. We find widespread violations of statistical self\-consistency, even among frontier models\. Across the models we study, statistical self\-consistency shows no apparent correlation with performance on standard capability benchmarks\. These findings suggest that statistical self\-consistency captures an underexplored dimension of model quality on which even frontier models have room to improve\.
One consequence of these violations is that statistically equivalent formulations can yield incompatible answers\. This has important implications for the robustness of existing alignment claims and for how we evaluate models’ ability to estimate uncertainty\. In particular, conclusions drawn at the aggregate level may not apply at a finer granularity\. Thus, an important open question is how to reliably evaluate LLMs in light of these inconsistencies\. A natural next step is to test whether optimizing for statistical self\-consistency can narrow the inconsistency gap we observe\. A perfectly accurate model necessarily satisfies these consistency criteria, yet satisfying them alone does not guarantee accuracy\. Violations could therefore provide a flexible, reference\-free training signal that complements conventional measures of accuracy\.
### Limitations
We evaluate statistical self\-consistency across multiple settings in both persona prompting and synthetic forecasting\. The criterion applies more broadly, and it remains to be seen how far our findings generalize beyond these settings\. We also want to emphasize that self\-consistency is a necessary but not sufficient condition for faithful conditional inference\. The availability of ACS reference data enables a separate analysis of distributional alignment, which reveals the macro fallacy\. Whether these alignment gains generalize to other domains is a promising direction for future work\. Alignment measurements also depend on design choices, including verbalized probability elicitation, post\-hoc normalization, and demographic splits\. Our synthetic forecasting examples demonstrate the applicability of consistency checks beyond survey\-based settings, but do not by themselves establish accuracy in real\-world forecasting or decision\-making\. It remains to evaluate self\-consistency across broader domains and richer conditioning structures\.
### Acknowledgments
We would like to thank Ana\-Andreea Stoica, Vivian Nastl, Florian Dorner, and Léo le Douarec for their valuable feedback on the manuscript\. We also thank the anonymous reviewers for their constructive comments on earlier versions of the paper\. PW was supported by the Max Planck ETH Center for Learning Systems\. PW also gratefully acknowledges G\-Research for providing financial support to attend NeurIPS in Sydney\. TKB was supported by an ETH AI Center Postdoctoral Fellowship\. CMD acknowledges the financial support of the Hector Foundation\.
### Reproducibility statement
To support reproducibility,[AppendixF](https://arxiv.org/html/2607.15277#A6)describes the experimental setup, data preprocessing, and evaluation procedures, while[AppendixH](https://arxiv.org/html/2607.15277#A8)provides the exact prompt templates\. The source code is publicly available at[https://github\.com/patrikwolf/statistical\_self\_consistency](https://github.com/patrikwolf/statistical_self_consistency)\.[AppendixD](https://arxiv.org/html/2607.15277#A4)states the assumptions underlying our theoretical results and provides their proofs\.
## References
- Argyleet al\.\(2023\)L\. P\. Argyle, E\. C\. Busby, N\. Fulda, J\. R\. Gubler, C\. Rytting, and D\. WingateOut of one, many: using language models to simulate human samples\.Political Analysis31\(3\),pp\. 337–351\.External Links:[Document](https://dx.doi.org/10.1017/pan.2023.2),ISSN 1476\-4989Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p2.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Artificial Analysis \(2026\)Artificial AnalysisArtificial analysis intelligence benchmarking methodology\.Note:Accessed: 2026\-07\-16[https://artificialanalysis\.ai/methodology/intelligence\-benchmarking](https://artificialanalysis.ai/methodology/intelligence-benchmarking)Cited by:[§F\.7](https://arxiv.org/html/2607.15277#A6.SS7.p2.1),[Figure 4](https://arxiv.org/html/2607.15277#S5.F4),[Figure 4](https://arxiv.org/html/2607.15277#S5.F4.6),[§6](https://arxiv.org/html/2607.15277#S6.SS0.SSS0.Px1.p1.1)\.
- Barrie and Cerina \(2026\)C\. Barrie and R\. CerinaSynthetic personas distort the structure of human belief systems\.SocArXiv\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Becket al\.\(2024\)T\. Beck, H\. Schuff, A\. Lauscher, and I\. GurevychSensitivity, performance, robustness: deconstructing the effect of sociodemographic prompting\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.15277#S3.SS2.SSS0.Px1.p1.1)\.
- Becker and Kohavi \(1996\)B\. Becker and R\. KohaviAdult\.Note:UCI Machine Learning RepositoryCited by:[§3\.2](https://arxiv.org/html/2607.15277#S3.SS2.p1.1)\.
- Bertsekas and Tsitsiklis \(2008\)D\. Bertsekas and J\. N\. TsitsiklisIntroduction to probability\.Vol\.1,Athena Scientific\.Cited by:[§1\.1](https://arxiv.org/html/2607.15277#S1.SS1.p1.1)\.
- Bisbeeet al\.\(2024\)J\. Bisbee, J\. D\. Clinton, C\. Dorff, B\. Kenkel, and J\. M\. LarsonSynthetic replacements for human survey data? The perils of large language models\.Political Analysis32\(4\),pp\. 401–416\.External Links:[Document](https://dx.doi.org/10.1017/pan.2024.5)Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.15277#S4.SS1.p2.1)\.
- Bottmanet al\.\(2026\)N\. Bottman, Y\. Liu, and K\. RichardsonOperadic consistency: a label\-free signal for compositional reasoning failures in LLMs\.External Links:[Link](https://arxiv.org/abs/2606.13649),2606\.13649Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
- Breimanet al\.\(1984\)L\. Breiman, J\. H\. Friedman, R\. A\. Olshen, and C\. J\. StoneClassification and regression trees\.1 edition,Chapman and Hall/CRC\(eng\)\.External Links:[Document](https://dx.doi.org/10.1201/9781315139470)Cited by:[§3\.1](https://arxiv.org/html/2607.15277#S3.SS1.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Bueninget al\.\(2026\)T\. K\. Buening, J\. Hübotter, B\. Pásztor, I\. Shenfeld, G\. Ramponi, and A\. KrauseAligning language models from user interactions\.InThird Conference on Language Modeling,Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2024\)X\. Chen, R\. A\. Chi, X\. Wang, and D\. ZhouPremise order matters in reasoning with large language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§5\.2](https://arxiv.org/html/2607.15277#S5.SS2.p1.1)\.
- Chlonet al\.\(2025\)L\. Chlon, S\. Rashidi, Z\. Khamis, and M\. M\. AwadaLLMs are Bayesian, in expectation, not in realization\.External Links:[Link](https://arxiv.org/abs/2507.11768),2507\.11768Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Cruzet al\.\(2024\)A\. F\. Cruz, M\. Hardt, and C\. Mendler\-DünnerEvaluating language models as risk scores\.InThe Thirty\-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§E\.2](https://arxiv.org/html/2607.15277#A5.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.15277#S1.p3.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.15277#S3.SS2.SSS0.Px1.p1.1)\.
- de Carvalhoet al\.\(2025\)G\. S\. P\. de Carvalho, M\. Abdulrahman, H\. Wang, S\. G\. Subramanian, M\. St\-Aubin, S\. O’Sullivan, L\. Wan, L\. Ricardez\-Sandoval, P\. Poupart, and A\. KristiadiSimplifying Bayesian optimization via in\-context direct optimum sampling\.External Links:[Link](https://arxiv.org/abs/2505.23913),2505\.23913Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Diaconis \(1988\)P\. DiaconisGroup representations in probability and statistics\.IMS Lecture Notes,Institute of Mathematical Statistics\.External Links:ISBN 9780940600140,LCCN 88082779Cited by:[§D\.1](https://arxiv.org/html/2607.15277#A4.SS1.p1.6)\.
- Dinget al\.\(2021\)F\. Ding, M\. Hardt, J\. Miller, and L\. SchmidtRetiring adult: new datasets for fair machine learning\.Advances in Neural Information Processing Systems34,pp\. 6478–6490\.Cited by:[§E\.2](https://arxiv.org/html/2607.15277#A5.SS2.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2607.15277#S3.SS2.p1.1)\.
- Dominguez\-Olmedoet al\.\(2024\)R\. Dominguez\-Olmedo, M\. Hardt, and C\. Mendler\-DünnerQuestioning the survey responses of large language models\.Advances in Neural Information Processing Systems37,pp\. 45850–45878\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Durmuset al\.\(2024\)E\. Durmus, K\. Nguyen, T\. Liao, N\. Schiefer, A\. Askell, A\. Bakhtin, C\. Chen, Z\. Hatfield\-Dodds, D\. Hernandez, N\. Joseph, L\. Lovitt, S\. McCandlish, O\. Sikder, A\. Tamkin, J\. Thamkul, J\. Kaplan, J\. Clark, and D\. GanguliTowards measuring the representation of subjective global opinions in language models\.InFirst Conference on Language Modeling,Cited by:[§E\.2](https://arxiv.org/html/2607.15277#A5.SS2.SSS0.Px2.p1.1)\.
- Elazaret al\.\(2021\)Y\. Elazar, N\. Kassner, S\. Ravfogel, A\. Ravichander, E\. Hovy, H\. Schütze, and Y\. GoldbergMeasuring and improving consistency in pretrained language models\.Transactions of the Association for Computational Linguistics9,pp\. 1012–1031\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
- Falcket al\.\(2024\)F\. Falck, Z\. Wang, and C\. C\. HolmesIs in\-context learning in large language models Bayesian? A martingale perspective\.InForty\-first International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Fluriet al\.\(2024\)L\. Fluri, D\. Paleka, and F\. TramèrEvaluating superhuman models with consistency checks\.In2024 IEEE Conference on Secure and Trustworthy Machine Learning \(SaTML\),Vol\.,pp\. 194–232\.External Links:[Document](https://dx.doi.org/10.1109/SaTML59370.2024.00017)Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.15277#S6.SS0.SSS0.Px2.p1.1)\.
- Guanet al\.\(2025\)B\. Guan, T\. Roosta, P\. Passban, and M\. RezagholizadehThe order effect: investigating prompt sensitivity to input order in LLMs\.External Links:2502\.04134,[Link](https://arxiv.org/abs/2502.04134)Cited by:[§5\.2](https://arxiv.org/html/2607.15277#S5.SS2.p1.1)\.
- Huet al\.\(2026\)T\. Hu, J\. Baumann, L\. Lupo, N\. Collier, D\. Hovy, and P\. RöttgerSimBench: benchmarking the ability of large language models to simulate human behaviors\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)X\. L\. Li, V\. Shrivastava, S\. Li, T\. Hashimoto, and P\. LiangBenchmarking and improving generator\-validator consistency of language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
- Lutzet al\.\(2025\)M\. Lutz, I\. Sen, G\. Ahnert, E\. Rogers, and M\. StrohmaierThe prompt makes the person\(a\): a systematic evaluation of sociodemographic persona prompting for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 23212–23237\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1261),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Maet al\.\(2025\)B\. Ma, B\. Yoztyurk, A\. Haensch, X\. Wang, M\. Herklotz, F\. Kreuter, B\. Plank, and M\. AßenmacherAlgorithmic fidelity of large language models in generating synthetic German public opinions: a case study\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 1785–1809\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.90),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Meisteret al\.\(2025\)N\. Meister, C\. Guestrin, and T\. HashimotoBenchmarking distributional alignment of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p2.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2607.15277#S4.p1.2)\.
- Mülleret al\.\(2023\)S\. Müller, M\. Feurer, N\. Hollmann, and F\. HutterPFNs4BO: in\-context learning for Bayesian optimization\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p1.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Palekaet al\.\(2025\)D\. Paleka, A\. P\. Sudhir, A\. Alvarez, V\. Bhat, A\. Shen, E\. Wang, and F\. TramèrConsistency checks for language model forecasters\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.15277#S6.SS0.SSS0.Px2.p1.1)\.
- Paruchuriet al\.\(2024\)A\. Paruchuri, J\. Garrison, S\. Liao, J\. B\. Hernandez, J\. Sunshine, T\. Althoff, X\. Liu, and D\. McDuffWhat are the odds? Language models are capable of probabilistic reasoning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 11712–11733\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
- Reuteret al\.\(2025\)A\. Reuter, T\. G\. J\. Rudner, V\. Fortuin, and D\. RügamerCan transformers learn full Bayesian inference in context?\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p1.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Santambrogio \(2015\)F\. SantambrogioOptimal transport for applied mathematicians: calculus of variations, pdes, and modeling\.1st edition,Progress in Nonlinear Differential Equations and Their Applications, 87,Springer International Publishing,Cham\(eng\)\.External Links:ISBN 3\-319\-20828\-4Cited by:[§C\.2](https://arxiv.org/html/2607.15277#A3.SS2.p2.2.1)\.
- Santurkaret al\.\(2023\)S\. Santurkar, E\. Durmus, F\. Ladhak, C\. Lee, P\. Liang, and T\. HashimotoWhose opinions do language models reflect?\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 29971–30004\.Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p2.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px2.p1.1)\.
- Schulhoffet al\.\(2025\)S\. Schulhoff, M\. Ilie, N\. Balepur, K\. Kahadze, A\. Liu, C\. Si, Y\. Li, A\. Gupta, H\. Han, S\. Schulhoff, P\. S\. Dulepet, S\. Vidyadhara, D\. Ki, S\. Agrawal, C\. Pham, G\. Kroiz, F\. Li, H\. Tao, A\. Srivastava, H\. D\. Costa, S\. Gupta, M\. L\. Rogers, I\. Goncearenco, G\. Sarli, I\. Galynker, D\. Peskoff, M\. Carpuat, J\. White, S\. Anadkat, A\. Hoyle, and P\. ResnikThe prompt report: a systematic survey of prompt engineering techniques\.External Links:2406\.06608,[Link](https://arxiv.org/abs/2406.06608)Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- U\.S\. Census Bureau \(2024\)U\.S\. Census BureauAmerican Community Survey \(ACS\) 1\-Year Public Use Microdata Sample\.Note:Accessed: 2026\-04\-01[https://www\.census\.gov/programs\-surveys/acs\.html](https://www.census.gov/programs-surveys/acs.html)Cited by:[§F\.1](https://arxiv.org/html/2607.15277#A6.SS1.p1.1)\.
- Wanget al\.\(2022\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
- Xieet al\.\(2022\)S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. MaAn explanation of in\-context learning as implicit Bayesian inference\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.15277#S1.p1.1),[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Q\. Yang, S\. Mahns, S\. Li, A\. Gu, J\. Wu, and H\. XuLLM\-as\-a\-prophet: understanding predictive intelligence with prophet arena\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2607.15277#S6.SS0.SSS0.Px2.p1.1)\.
- Zhu and Griffiths \(2024\)J\. Zhu and T\. L\. GriffithsIncoherent probability judgments in large language models\.arXiv preprint arXiv:2401\.16646\.Cited by:[§2](https://arxiv.org/html/2607.15277#S2.SS0.SSS0.Px3.p1.1)\.
## Appendices
## Contents
## Appendix AIllustrative reconstruction across tree levels
[Figure5](https://arxiv.org/html/2607.15277#A1.F5)provides a schematic illustration of the evaluation procedure introduced in[Section3](https://arxiv.org/html/2607.15277#S3)\. Starting from a base population, we partition the population into mutually exclusive and exhaustive subgroups, elicit conditional estimates and subgroup priors from the LLM, and reconstruct the population\-level estimate using the law of total probability\. The figure shows simplified prompts and intermediate quantities involved in the reconstruction\.
Level 0: rootIncome0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 1:age∉\[31,68\]\\notin\[31,68\]Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}young & oldLevel 1:age∈\[31,68\]\\in\[31,68\]Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}working ageConditional prompt:Given a random person in the U\.S\.,estimate the probability that this person’s annual income falls into each income bin\.Conditional prompt:Given a random person in the U\.S\. aged below 31 or above 68,estimate the probability that this person’s annual income falls into each income bin\.Conditional prompt:Given a random person in the U\.S\. aged between 31 and 68,estimate the probability that this person’s annual income falls into each income bin\.Prior prompt:What share of the U\.S\. population belongs to each of the two age groups? Return probabilities summing to one\.Level\-1 aggregateIncome0\.00\.00\.20\.2ℙ\\mathbb\{P\}Self\-consistency discrepancyIncome bins\[−11\.5k,1\)\[\-11\.5\\mathrm\{k\},1\)USD\[1,25k\)\[1,25\\mathrm\{k\}\)USD\[25k,60k\)\[25\\mathrm\{k\},60\\mathrm\{k\}\)USD\[60k,1849k\]\[60\\mathrm\{k\},1849\\mathrm\{k\}\]USD
Figure 5:Illustrative reconstruction of an aggregate estimate from subgroup estimates\.We show one refinement step of the binary conditioning tree for the ACS income\-distribution task\. At level 0, the LLM directly estimates the income distribution of the base population\. At level 1, the population is partitioned by age into two mutually exclusive and exhaustive subgroups\. The LLM estimates the conditional income distribution for each subgroup from its verbalized description\. A separate prior\-elicitation prompt estimates the relative sizes of the two subgroups\. The resulting LLM\-estimated priors are used to combine the conditional distributions into the level\-1 aggregate according to the law of total probability\. The normalized Wasserstein\-11distance between the direct root estimateT^h\(𝒳\)\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\)and the reconstructed aggregateT^h,agg\(𝒳\)\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\}\)quantifies the corresponding violation of statistical self\-consistency\. All displayed prompts are simplified for readability\.
## Appendix BDisentangling conditionals and priors
Deviations in the alignment error across levels in[Figure2](https://arxiv.org/html/2607.15277#S4.F2)indicate that model estimates fail to compose consistently under the law of total probability\. The concave shape of the aggregation gain suggests that the models’ ability to represent the population improves with depth until it starts declining as the partition becomes more fine\-grained\. In this appendix, we trace where these errors come from\. The reconstructed estimateT^τ,agg\(ℓ\)\(𝒳\)\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}^\{\(\\ell\)\}\(\\mathcal\{X\}\)combines two ingredients elicited from the model: the subgroup conditionalsT^τ\(𝒮\)\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)and the subgroup priorsγ^𝒮\\hat\{\\gamma\}\_\{\\mathcal\{S\}\}used to weight them\. Since the aggregate alignment error conflates both, we examine the quality of the two components separately\. The survey data makes this decomposition possible because it provides reference statistics for each subgroup induced by the binary conditioning tree\. We can therefore go beyond evaluating whether reconstructed population\-level estimates align with the survey\-based ground truth, and directly analyze how well the LLM estimates the conditional quantities at different levels of specificity\. In particular, we first study an oracle\-prior regime in which subgroup estimates are aggregated using ground\-truth survey priors, thereby isolating the contribution of the LLM\-estimated conditional probabilities to the macro fallacy\. We then compare the LLM\-estimated priors themselves against the survey\-based subgroup proportions to assess how errors in population weighting evolve as the partition becomes finer\.
### B\.1Conditional alignment across tree levels
The aggregate alignment error in \([5](https://arxiv.org/html/2607.15277#S4.E5)\) is computed from both LLM\-estimated subgroup conditionalsT^τ\(𝒮\)\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)and LLM\-estimated subgroup priorsγ^𝒮\\hat\{\\gamma\}\_\{\\mathcal\{S\}\}, thereby conflating two sources of error\. To isolate the error caused by the conditionals, we replace the LLM\-estimated priors with ground\-truth subgroup priorsγ𝒮\\gamma\_\{\\mathcal\{S\}\}obtained from the survey data\. This yields the conditional alignment error
CAErr\(ℓ\)≜\|Tτ\(𝒳\)−∑𝒮∈𝒫ℓ\(𝒳\)γ𝒮T^τ\(𝒮\)\|\.\\text\{CAErr\}\(\\ell\)\\triangleq\\bigg\|\\,T\_\{\\tau\}\(\\mathcal\{X\}\)\-\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\gamma\_\{\\mathcal\{S\}\}\\,\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)\\,\\bigg\|\.\(13\)This oracle\-prior regime removes uncertainty about the population composition and measures how accurately the LLM\-estimated conditionals aggregate to the ground\-truth population statistic\.
0123456Level−\-40%−\-20%0%20%40%60%80%Aggregation gainτ=100\\tau=100USDτ=1k\\tau=1\\mathrm\{k\}USDτ=10k\\tau=10\\mathrm\{k\}USDτ=20k\\tau=20\\mathrm\{k\}USDτ=40k\\tau=40\\mathrm\{k\}USDτ=60k\\tau=60\\mathrm\{k\}USDτ=80k\\tau=80\\mathrm\{k\}USDFigure 6:Conditional aggregation gain using ground\-truth priors\.On average across the considered income thresholds, reconstructing population\-level estimates from increasingly fine\-grained subgroup estimates using ground\-truth subgroup priors improves alignment relative to direct prompting across all considered partitions\. Error bars show 90 % confidence intervals computed by bootstrapping with 1000 samples\.[Figure6](https://arxiv.org/html/2607.15277#A2.F6)reports the corresponding conditional aggregation gain,1−CAErr\(ℓ\)/CAErr\(0\)1\-\\text\{CAErr\}\(\\ell\)/\\text\{CAErr\}\(0\), across income thresholds\. Positive values indicate that reconstructing the aggregate from subgroup estimates improves upon direct prompting\. Over the first few partition depths, the reconstructed estimates are generally better aligned with the ACS reference statistic than the root\-level estimate\. At greater depths, however, this advantage can diminish or even reverse\. Together with the node\-wise results, this pattern shows that the initial aggregation gains are already present in the LLM’s conditional estimates, indicating that subgroup\-conditioned estimates contain useful population\-level information that is not reliably elicited by direct prompting\.
#### Categorical income distributions\.
We next repeat the oracle\-prior aggregation experiment for a categorical income target\. Instead of estimating the probability of exceeding a fixed threshold, the model is asked to estimate the full distribution over income bins\. To quantify alignment for such distribution\-valued targets, we generalize the conditional alignment error in \([13](https://arxiv.org/html/2607.15277#A2.E13)\) to
CAErr\(ℓ\)≜𝒲¯1\(Th\(𝒳\),∑𝒮∈𝒫ℓ\(𝒳\)γ𝒮T^h\(𝒮\)\),\\text\{CAErr\}\(\\ell\)\\triangleq\\overline\{\\mathcal\{W\}\}\_\{1\}\\bigg\(T\_\{h\}\(\\mathcal\{X\}\),\\,\{\\sum\}\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\gamma\_\{\\mathcal\{S\}\}\\,\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)\\bigg\),\(14\)where𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}denotes the normalized Wasserstein\-11distance\. This definition strictly generalizes the thresholded case\. A threshold event\{Y\>τ\}\\\{Y\>\\tau\\\}induces Bernoulli distributions with atoms at unit distance, for which the Wasserstein\-11distance reduces to the absolute difference of tail probabilities, recovering \([13](https://arxiv.org/html/2607.15277#A2.E13)\)\.
[Figure7](https://arxiv.org/html/2607.15277#A2.F7)shows the reconstructed population\-level income distributions at different levels of the BCT, along with the corresponding errors \([14](https://arxiv.org/html/2607.15277#A2.E14)\)\. The root\-level estimate corresponds to direct elicitation of the aggregate income distribution, whereas levels one to three reconstruct this estimate from increasingly refined LLM\-estimated subgroup distributions, using ACS ground\-truth subgroup priors as weights\. The tree structure is detailed in[SectionF\.3](https://arxiv.org/html/2607.15277#A6.SS3)\. Consistent with the thresholded results, the reconstructed distributions are better aligned with the ACS reference distribution than the direct root\-level estimate\. This shows that the benefit of subgroup aggregation is not limited to binary threshold events\. Further, we observe that the level\-3 reconstruction is slightly less accurate than the level\-2 reconstruction, suggesting that the gains from refinement are eventually offset by noisier estimates for very fine\-grained subgroups\.
CAErr\(0\)=0\.14\\mathrm\{CAErr\}\(0\)=0\.14Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 0 \(Root\)CAErr\(1\)=0\.04\\mathrm\{CAErr\}\(1\)=0\.04Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 1CAErr\(2\)=0\.03\\mathrm\{CAErr\}\(2\)=0\.03Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 2CAErr\(3\)=0\.03\\mathrm\{CAErr\}\(3\)=0\.03Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 3ACSLLM\[−11\.5k,1\)\[\-11\.5\\mathrm\{k\},1\)USD\[1,25k\)\[1,25\\mathrm\{k\}\)USD\[25k,60k\)\[25\\mathrm\{k\},60\\mathrm\{k\}\)USD\[60k,1849k\]\[60\\mathrm\{k\},1849\\mathrm\{k\}\]USD 000\.050\.050\.100\.100\.150\.150\.200\.20
Figure 7:Reconstructed aggregate income distributions with ground\-truth priors\.We reconstruct the population\-level income distribution at each tree level by aggregating LLM\-estimated subgroup distributions using ACS subgroup priors\. Solid bars show the survey\-based aggregate distribution, while hatched bars show the aggregate reconstructed from LLM\-estimated subgroup conditionals\. The colored border indicates the conditional alignment errorCAErr\(ℓ\)\\mathrm\{CAErr\}\(\\ell\), with red denoting larger errors and green denoting smaller errors\.
### B\.2LLM\-estimated priors
Deeper partitions require estimates for a growing number of increasingly specific subgroup priors\. Since aggregation gains persist to greater depths when oracle priors are used, errors in the LLM\-estimated priors appear to increasingly offset the benefits of finer\-grained conditioning\. We investigate this possibility by comparing the estimated subgroup priors with their survey\-based counterparts\.
Following the level\-wise elicitation protocol described in the main text and detailed in[SectionF\.2](https://arxiv.org/html/2607.15277#A6.SS2), letγℓ\\gamma\_\{\\ell\}denote the survey\-based distribution over the subgroups at tree levelℓ\\ell, and letγ^ℓ\\hat\{\\gamma\}\_\{\\ell\}denote the corresponding LLM\-estimated distribution\. We quantify their discrepancy using the total variation \(TV\) distance,
TV\(γ^ℓ,γℓ\)=12∑𝒮∈𝒫ℓ\|γ^ℓ\(𝒮\)−γℓ\(𝒮\)\|,\\operatorname\{TV\}\(\\hat\{\\gamma\}\_\{\\ell\},\\gamma\_\{\\ell\}\)=\\frac\{1\}\{2\}\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\\hat\{\\gamma\}\_\{\\ell\}\(\\mathcal\{S\}\)\-\\gamma\_\{\\ell\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\},\(15\)where𝒫ℓ\\mathcal\{P\}\_\{\\ell\}denotes the partition induced by levelℓ\\ell\. Because TV distance lies in\[0,1\]\[0,1\], it provides a normalized measure of prior misalignment that remains comparable across tree levels, even as the number of subgroups grows with depth\.
1234Level0\.00\.00\.10\.10\.20\.20\.30\.30\.40\.40\.50\.5TV distance1234Level0\.00\.00\.10\.10\.20\.20\.30\.30\.40\.40\.50\.50\.60\.6TV distanceGPT\-5\.4GPT\-4o mini3\.1 Pro3 FlashSonnet 4\.6Average
\(a\)BCT with splits:AGEP,COW,WKHP,OCCP\(b\)BCT with splits:COW,MAR,ESR,AGEP
Figure 8:LLM\-estimated priors become less aligned at deeper tree levels\.For each tree level, we compare the LLM\-estimated distribution over subgroups with the survey\-based ground\-truth prior distribution from the ACS data using total variation \(TV\) distance\. Across models, prior alignment deteriorates as the tree becomes more fine\-grained\. Error bars show 90 % confidence intervals computed by bootstrapping with 1000 samples\.[Figure8](https://arxiv.org/html/2607.15277#A2.F8)shows that LLM\-estimated priors become less aligned with the survey\-based ground truth as the tree becomes more fine\-grained\. This trend is consistent across the evaluated models\. While coarse population splits are estimated relatively accurately, errors increase as the model is asked to assign probability mass across more specific subpopulations\. This pattern indicates that estimating the population composition of fine\-grained demographic partitions remains challenging for current models and helps explain why empirical aggregation gains diminish at greater depths\. Aggregation therefore entails a trade\-off between improved conditional specificity and more difficult population weighting\.
## Appendix CExtensions of self\-consistency checks
We provide additional details for the self\-consistency checks introduced in[Section5](https://arxiv.org/html/2607.15277#S5)\. We first discuss the choice of discrepancy metric for binary and categorical distributions\. We then interpret self\-consistency as operations on a binary conditioning tree\. Finally, we derive the expected variance contraction criterion from the law of total variance\.
### C\.1Discrepancy metrics for outcome distributions
The self\-consistency checks in[Definitions1](https://arxiv.org/html/2607.15277#Thmtheorem1)and[3](https://arxiv.org/html/2607.15277#Thmtheorem3)are stated for a general discrepancy metric\. This allows the same aggregation principle to be applied to different types of elicited model outputs\. In this subsection, we discuss how the distance metric can be defined in three canonical examples\.
#### Binary distributions\.
In the ACS income setting of[Sections3](https://arxiv.org/html/2607.15277#S3)and[4](https://arxiv.org/html/2607.15277#S4), the target event isY\>τY\>\\tau, whereYYdenotes yearly income in USD andτ\\tauis an income threshold\. The corresponding tail probability can be written as a special case of our general target functional:
Tτ\(𝒮\)≜ℙ\(Y\>τ\|X∈𝒮\)=𝔼\[h\(Y\)\|X∈𝒮\]withh:Y↦𝟙\{𝕐\>τ\}\.T\_\{\\tau\}\(\\mathcal\{S\}\)\\triangleq\\mathbb\{P\}\(Y\>\\tau\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)=\\mathbb\{E\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\[$\}\}\}h\(Y\)\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\]$\}\}\}\\qquad\\text\{with\}\\qquad h:Y\\mapsto\\mathbbold\{1\}\\\{Y\>\\tau\\\}\.\(16\)Thus, each estimate corresponds to the success probability of a Bernoulli distribution indicating whether an individual in subpopulation𝒮\\mathcal\{S\}has income above the threshold\. For this binary target, the natural discrepancy between a direct estimateT^τ\(𝒮\)\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)and the aggregated estimateT^τ,agg\(𝒮\)\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)from \([4](https://arxiv.org/html/2607.15277#S3.E4)\) is the absolute difference
dist\(T^τ\(𝒮\),T^τ,agg\(𝒮\)\)=\|T^τ\(𝒮\)−T^τ,agg\(𝒮\)\|\.\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}=\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\\hat\{T\}\_\{\\tau\}\(\\mathcal\{S\}\)\-\\hat\{T\}\_\{\\tau,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\.\(17\)For Bernoulli distributions, the absolute difference between success probabilities coincides with the \(normalized\) Wasserstein\-11distance on the ordered support\{0,1\}\\\{0,1\\\}\.
#### Ordered categorical outcomes\.
More generally, the outcome of interest may be categorical, with support\{1,…,K\}\\\{1,\\dots,K\\\}\. In this case, we useT^h\(𝒮\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)to denote the elicited conditional distribution over theKKcategories conditional on membership in𝒮\\mathcal\{S\}\. Similarly,T^h,agg\(𝒮\)\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)denotes the corresponding prior\-weighted aggregate\. The consistency error can then be defined by using an appropriate distance between probability distributions\. For ordinal response options, we use the normalized Wasserstein\-11distance,
dist\(T^h\(𝒮\),T^h,agg\(𝒮\)\)=𝒲¯1\(T^h\(𝒮\),T^h,agg\(𝒮\)\)≜1K−1∑k=1K−1\|F\(k\)−Fagg\(k\)\|,\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}=\\overline\{\\mathcal\{W\}\}\_\{1\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\\triangleq\\frac\{1\}\{K\-1\}\\sum\_\{k=1\}^\{K\-1\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}F\(k\)\-F\_\{\\textnormal\{agg\}\}\(k\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\},\(18\)whereFFandFaggF\_\{\\textnormal\{agg\}\}denote the corresponding cumulative distribution functions ofT^h\(𝒮\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)andT^h,agg\(𝒮\)\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\), respectively\. This metric accounts for how far probability mass must be shifted along the response order\. The normalization maps the distance to\[0,1\]\[0,1\], making discrepancies comparable across questions with different numbers of response options\.
#### Nominal categorical outcomes\.
For nominal response sets, the category labels have no meaningful order\. We therefore use a permutation\-invariant discrepancy, such as total variation distance:
dist\(T^h\(𝒮\),T^h,agg\(𝒮\)\)=TV\(T^h\(𝒮\),T^h,agg\(𝒮\)\)\.\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}=\\operatorname\{TV\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\.\(19\)If both distributions are represented as probability vectorsp,pagg∈ΔK−1p,p\_\{\\textnormal\{agg\}\}\\in\\Delta^\{K\-1\}, this is given by
TV\(T^h\(𝒮\),T^h,agg\(𝒮\)\)=12∑k=1K\|p\[k\]−pagg\[k\]\|\.\\operatorname\{TV\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}=\\frac\{1\}\{2\}\\sum\_\{k=1\}^\{K\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}p\[k\]\-p\_\{\\textnormal\{agg\}\}\[k\]\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\.\(20\)The factor1/21/2normalizes the distance to lie in\[0,1\]\[0,1\]\.
### C\.2Wasserstein distance on ordered discrete outcomes
This subsection recalls the closed\-form expression of the Wasserstein\-11distance on ordered discrete supports and explains the normalization used to compare distributions with different numbers of categories\.
###### Lemma 6\.
Letμ\\muandν\\nube discrete probability measures on the ordered support\{1,…,K\}\\\{1,\\dots,K\\\}, equipped with the metricd\(i,j\)=\|i−j\|d\(i,j\)=\|i\-j\|, and letFμF\_\{\\mu\}andFνF\_\{\\nu\}denote their cumulative distribution functions\. Then
𝒲1\(μ,ν\)=∑k=1K−1\|Fμ\(k\)−Fν\(k\)\|\.\\mathcal\{W\}\_\{1\}\(\\mu,\\nu\)=\\sum\_\{k=1\}^\{K\-1\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}F\_\{\\mu\}\(k\)\-F\_\{\\nu\}\(k\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\.\(21\)
###### Proof\.
For probability measures on the real line, the Wasserstein\-11distance admits the classical closed form
𝒲1\(μ,ν\)=∫ℝ\|Fμ\(t\)−Fν\(t\)\|𝑑t,\\mathcal\{W\}\_\{1\}\(\\mu,\\nu\)=\\int\_\{\\mathbb\{R\}\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\|$\}\}\}F\_\{\\mu\}\(t\)\-F\_\{\\nu\}\(t\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\|$\}\}\}\\;\\mathrm\{d\}t,\(22\)see\[[Santambrogio, 2015](https://arxiv.org/html/2607.15277#bib.bib26), Proposition 2\.17\]\. Sinceμ\\muandν\\nuare supported on\{1,…,K\}\\\{1,\\dots,K\\\}, the integrand in equation \([22](https://arxiv.org/html/2607.15277#A3.E22)\) vanishes outside\[1,K\)\[1,K\)and is constant on each interval\[k,k\+1\)\[k,k\+1\)fork∈\[K−1\]k\\in\[K\-1\], where it equals\|Fμ\(k\)−Fν\(k\)\|\|F\_\{\\mu\}\(k\)\-F\_\{\\nu\}\(k\)\|\. Integrating over the intervals\[k,k\+1\)\[k,k\+1\)for allk∈\[K−1\]k\\in\[K\-1\]yields the result in equation \([21](https://arxiv.org/html/2607.15277#A3.E21)\)\. ∎
#### Normalization\.
The Wasserstein\-11distance on an ordered support, as introduced in[Lemma6](https://arxiv.org/html/2607.15277#Thmtheorem6), depends on the size of the support\. In particular, for probability measuresμ,ν\\mu,\\nusupported on\{1,…,K\}\\\{1,\\dots,K\\\}withK≥2K\\geq 2and metricd\(i,j\)=\|i−j\|d\(i,j\)=\|i\-j\|, the maximum possible distance isK−1K\-1, attained when all mass is placed on opposite endpoints\. Hence, raw Wasserstein distances are not directly comparable across multiple\-choice questions with different numbers of answer options\. We therefore use the normalized Wasserstein\-11distance
𝒲¯1\(μ,ν\)≜1K−1𝒲1\(μ,ν\)=1K−1∑k=1K−1\|Fμ\(k\)−Fν\(k\)\|\.\\overline\{\\mathcal\{W\}\}\_\{1\}\(\\mu,\\nu\)\\triangleq\\frac\{1\}\{K\-1\}\\,\\mathcal\{W\}\_\{1\}\(\\mu,\\nu\)=\\frac\{1\}\{K\-1\}\\sum\_\{k=1\}^\{K\-1\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}F\_\{\\mu\}\(k\)\-F\_\{\\nu\}\(k\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\.\(23\)This normalization maps the distance to\[0,1\]\[0,1\]independently of the number of ordered categories, with00indicating identical distributions and11indicating maximal separation on the support\.
#### Bernoulli distributions\.
For Bernoulli distributionsμ=Ber\(p\)\\mu=\\operatorname\{Ber\}\(p\)andν=Ber\(q\)\\nu=\\operatorname\{Ber\}\(q\)on the ordered support\{0,1\}\\\{0,1\\\}, equation \([21](https://arxiv.org/html/2607.15277#A3.E21)\) contains a single cumulative difference and therefore yields
𝒲1\(Ber\(p\),Ber\(q\)\)=\|\(1−p\)−\(1−q\)\|=\|p−q\|\.\\mathcal\{W\}\_\{1\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\operatorname\{Ber\}\(p\),\\operatorname\{Ber\}\(q\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}=\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}\(1\-p\)\-\(1\-q\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\|$\}\}\}=\|p\-q\|\.\(24\)Thus, the absolute error used in our binary experiments is exactly the Wasserstein\-11distance between the corresponding Bernoulli distributions\.
### C\.3Tree operations
The self\-consistency checks introduced in[Section5](https://arxiv.org/html/2607.15277#S5)admit a natural interpretation in terms of operations on binary conditioning trees\. Split consistency concerns the aggregation relations between parent and child nodes, whereas order consistency compares equivalent conditioning events obtained under different orders of the tree levels\. Interchanging two adjacent levels illustrates how these two requirements complement one another\.
#### Split consistency\.
Consider a node representing a conditioning event𝒮\\mathcal\{S\}and an admissible splitting attributeAkA\_\{k\}\. Refining𝒮\\mathcal\{S\}according to the two values ofAkA\_\{k\}yields the child events
𝒮k,0=𝒮∩\{Ak=0\}and𝒮k,1=𝒮∩\{Ak=1\},\\mathcal\{S\}\_\{k,0\}=\\mathcal\{S\}\\cap\\\{A\_\{k\}=0\\\}\\qquad\\text\{and\}\\qquad\\mathcal\{S\}\_\{k,1\}=\\mathcal\{S\}\\cap\\\{A\_\{k\}=1\\\},\(25\)which form a partition of𝒮\\mathcal\{S\}\. Split consistency requires the estimate elicited directly at the parent node to agree, up to the prescribed tolerance, with the prior\-weighted aggregate of the estimates at its children\. In[Figure9](https://arxiv.org/html/2607.15277#A3.F9), this corresponds to reconstructing the orange parent node from the two red child nodes\. Thus, every parent–child relation in a conditioning tree induces a local split consistency check\.
rootfemalefemaleyoungfemaleoldmalemaleyoungmaleoldrootyoungyoungfemaleyoungmaleoldoldfemaleoldmaleFigure 9:Tree interpretation of split and order consistency\.Within each tree, a parent estimate should agree with the prior\-weighted aggregate of its two child estimates\. Interchanging two adjacent levels preserves the leaf events but changes both the order in which their constraints are presented and the intermediate conditioning events exposed by the tree\.A fixed conditioning tree exposes only those refinements associated with its prescribed order of split attributes\. Other attributes may nevertheless define equally valid next splits of the same conditioning event\. For example, after interchanging two adjacent levels in[Figure9](https://arxiv.org/html/2607.15277#A3.F9), the pink node in the tree on the right is obtained by aggregating the two blue nodes shown in the tree on the left\. Because these blue nodes belong to different subtrees of the original tree, this parent–child relation is not exposed there\. This observation motivates defining the split consistency score in[Definition2](https://arxiv.org/html/2607.15277#Thmtheorem2)over all pairs of conditioning events and admissible next splits, rather than only over the parent–child relations of a single fixed tree\.
#### Order consistency\.
Interchanging tree levels also changes the order in which the corresponding attribute constraints are presented to the model\. For example, the tree on the left of[Figure9](https://arxiv.org/html/2607.15277#A3.F9)first splits by sex and then by age, whereas the tree on the right first splits by age and then by sex\. Nevertheless, corresponding leaves define the same conditioning events\. Order consistency therefore requires the direct estimates elicited for corresponding leaves to agree, up to the prescribed tolerance\.
### C\.4Beyond binary partitions
Throughout the paper, we construct partitions using binary conditioning trees\. This is a design choice rather than a conceptual restriction\. The law of total probability in \([2](https://arxiv.org/html/2607.15277#S1.E2)\) holds for any valid partition of the population\. In particular, suppose a subpopulation𝒮\\mathcal\{S\}is partitioned intoKKmutually disjoint and jointly exhaustive child subpopulations𝒮1,…,𝒮K\\mathcal\{S\}\_\{1\},\\dots,\\mathcal\{S\}\_\{K\}\. The corresponding aggregate estimate is
T^h,agg\(𝒮\)=∑k=1Kγ^kT^h\(𝒮k\),\\hat\{T\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{S\}\)=\\sum\_\{k=1\}^\{K\}\\hat\{\\gamma\}\_\{k\}\\kern 1\.00006pt\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{k\}\),\(26\)whereγ^k\\hat\{\\gamma\}\_\{k\}denotes the estimated conditional prevalence of𝒮k\\mathcal\{S\}\_\{k\}within𝒮\\mathcal\{S\}\. The split consistency check in[Definition1](https://arxiv.org/html/2607.15277#Thmtheorem1)extends directly by comparing thisKK\-term aggregate with the direct estimateT^h\(𝒮\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)\.
We nevertheless adopt binary trees for two practical reasons\. First, binary splits do not restrict the granularity of the analysis\. AnyKK\-way partition can be represented by a sequence ofK−1K\-1recursive binary splits, so binary trees generate arbitrarily fine partitions of the base population\. Second, binary splits simplify the construction of valid partitions\. Defining one child as the complement of the other within the parent guarantees by construction that every split is mutually exclusive and jointly exhaustive, as required by the law of total probability\. This makes it straightforward to build valid conditioning trees at varying levels of granularity\.
### C\.5Variance contraction
The self\-consistency checks in[Section5](https://arxiv.org/html/2607.15277#S5)constrain elicited estimates through first\-moment identities\. The law of total probability prescribes exactly how conditional estimates must aggregate across a partition\. The same reasoning extends to higher\-order moments of the outcome distribution\. In this subsection, we illustrate this with an auxiliary check derived from the law of total variance\. In particular, we reduce the law of total variance to a one\-sided constraint\. Refining a subpopulation can only reduce the outcome variance in expectation, since the variance of the conditional means across the children is non\-negative\.
###### Definition 7\(Expected variance contraction\)\.
Given a tree𝒯\\mathcal\{T\}, we say a model satisfies expected variance contraction if for every internal nodevvit holds that
𝕍^\(𝒮v\)≥γ^L𝕍^\(𝒮v,L\)\+γ^R𝕍^\(𝒮v,R\),\\hat\{\\mathbb\{V\}\}\(\\mathcal\{S\}\_\{v\}\)\\geq\\hat\{\\gamma\}\_\{L\}\\hat\{\\mathbb\{V\}\}\(\\mathcal\{S\}\_\{v,L\}\)\+\\hat\{\\gamma\}\_\{R\}\\hat\{\\mathbb\{V\}\}\(\\mathcal\{S\}\_\{v,R\}\),\(27\)where𝕍^\(∙\)\\hat\{\\mathbb\{V\}\}\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\)denotes the variance implied by the model’s estimated \(conditional\) distribution\.
We now derive the expected variance contraction condition used in[Definition7](https://arxiv.org/html/2607.15277#Thmtheorem7)from the law of total variance\. For two random variablesYYandAA, the law of total variance states that
𝕍\(Y\)=𝔼\[𝕍\(Y\|A\)\]\+𝕍\(𝔼\[Y\|A\]\)\.\\mathbb\{V\}\(Y\)=\\mathbb\{E\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\[$\}\}\}\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\]$\}\}\}\+\\mathbb\{V\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\mathbb\{E\}\[Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\]\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}\.\(28\)Since the second term is non\-negative, conditioning onAAcan only reduce variance in expectation:
𝕍\(Y\)≥𝔼\[𝕍\(Y\|A\)\]\.\\mathbb\{V\}\(Y\)\\geq\\mathbb\{E\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\[$\}\}\}\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptA\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\]$\}\}\}\.\(29\)Importantly, this is an averaged statement over the values ofAA; it does not imply that the conditional variance decreases for every realization ofAAindividually\. We now instantiate \([29](https://arxiv.org/html/2607.15277#A3.E29)\) at an internal node of a binary conditioning tree\. Let𝒮\\mathcal\{S\}be a parent subpopulation with children𝒮L\\mathcal\{S\}\_\{L\}and𝒮R\\mathcal\{S\}\_\{R\}, and letAAencode child membership within𝒮\\mathcal\{S\}\. That is,A=LA=Lcorresponds toX∈𝒮LX\\in\\mathcal\{S\}\_\{L\}andA=RA=Rcorresponds toX∈𝒮RX\\in\\mathcal\{S\}\_\{R\}, conditional onX∈𝒮X\\in\\mathcal\{S\}\. Applying \([29](https://arxiv.org/html/2607.15277#A3.E29)\) within the parent subpopulation𝒮\\mathcal\{S\}yields
𝕍\(Y\|X∈𝒮\)≥γL𝕍\(Y\|X∈𝒮L\)\+γR𝕍\(Y\|X∈𝒮R\),\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)\\geq\\gamma\_\{L\}\\,\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\_\{L\}\)\+\\gamma\_\{R\}\\,\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\_\{R\}\),\(30\)whereγL=ℙ\(X∈𝒮L\|X∈𝒮\)\\gamma\_\{L\}=\\mathbb\{P\}\(X\\in\\mathcal\{S\}\_\{L\}\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)andγR=ℙ\(X∈𝒮R\|X∈𝒮\)\\gamma\_\{R\}=\\mathbb\{P\}\(X\\in\\mathcal\{S\}\_\{R\}\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)are the child priors within the parent\. Thus, if the LLM estimates form a self\-consistent system of conditional distributions, the same inequality should hold for the corresponding LLM\-estimated variances, denoted by𝕍^\\hat\{\\mathbb\{V\}\}\. In our check, we replace the population variances and child priors by the corresponding model\-induced quantities, yielding the condition stated in[Definition7](https://arxiv.org/html/2607.15277#Thmtheorem7)\.
## Appendix DUniform consistency guarantees
This appendix develops the technical results underlying our uniform consistency guarantee\. We first establish auxiliary bounds and then combine them to control aggregate estimates across tree levels and tree structures\.
### D\.1Notation and auxiliary results
Throughout this appendix, let𝒯\\mathcal\{T\}be a binary conditioning tree of depthL≤dL\\leq d, induced by𝒜=\{A1,…,Ad\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{d\}\\\}and rooted at the base population𝒳\\mathcal\{X\}\. We index constraints by their attribute, so that constraintiiis the constraint imposed onAiA\_\{i\}, and we index positions by\{1,…,d\}\\\{1,\\dots,d\\\}likewise\. A*constraint ordering*is then a bijectionπ:\{1,…,d\}→\{1,…,d\}\\pi:\\\{1,\\dots,d\\\}\\to\\\{1,\\dots,d\\\}, whereπ\(i\)\\pi\(i\)is the position assigned to constraintii\. The canonical ordering is denoted byπ¯\\overline\{\\pi\}\. For a conditioning event𝒮\\mathcal\{S\}, we writeπ𝒮\\pi\_\{\\mathcal\{S\}\}andπ¯𝒮\\overline\{\\pi\}\_\{\\mathcal\{S\}\}for the restrictions of these orderings to the constraints defining𝒮\\mathcal\{S\}\. Forr∈\{0,…,L\}r\\in\\\{0,\\dots,L\\\}, letℒr\(𝒯\)\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)denote the nodes at levelrrand letγ^r\\hat\{\\gamma\}\_\{r\}be the LLM\-estimated prior distribution onℒr\(𝒯\)\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)\. Forv∈ℒr\(𝒯\)v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\), denote its conditioning event by𝒮v\\mathcal\{S\}\_\{v\}and writeπv≜π𝒮v\\pi\_\{v\}\\triangleq\\pi\_\{\\mathcal\{S\}\_\{v\}\}\. We define the aggregate estimate obtained from levelrras
T^h,agg\(r\)\(𝒳,π\)≜∑v∈ℒr\(𝒯\)γ^r\(v\)T^h\(𝒮v,πv\),\\hat\{T\}^\{\(r\)\}\_\{h,\\mathrm\{agg\}\}\(\\mathcal\{X\};\\pi\)\\triangleq\\sum\_\{v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{r\}\(v\)\\,\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\pi\_\{v\}\),\(31\)which atr=0r=0reduces to the direct root estimateT^h\(𝒳\)\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\), independently ofπ\\pi\. To compare aggregates at adjacent levels, we marginalize the child\-level prior onto the parent\-level partition\. Specifically, since every level\-\(r\+1\)\(r\+1\)node has a unique parent at levelrr, the priorγ^r\+1\\hat\{\\gamma\}\_\{r\+1\}induces a prior overℒr\(𝒯\)\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)by summing the masses of each parent’s children:
γ^r\+1↑\(v\)≜∑b∈\{0,1\}γ^r\+1\(v,b\)\.\\hat\{\\gamma\}^\{\\uparrow\}\_\{r\+1\}\(v\)\\triangleq\\sum\_\{b\\in\\\{0,1\\\}\}\\hat\{\\gamma\}\_\{r\+1\}\(v,b\)\.\(32\)Because the priors are elicited separately at each level,γ^r\\hat\{\\gamma\}\_\{r\}need not agree with the marginal ofγ^r\+1\\hat\{\\gamma\}\_\{r\+1\}\. We quantify their deviation by
δr\(𝒯\)≜TV\(γ^r,γ^r\+1↑\)\.\\delta\_\{r\}\(\\mathcal\{T\}\)\\triangleq\\operatorname\{TV\}\(\\hat\{\\gamma\}\_\{r\},\\hat\{\\gamma\}^\{\\uparrow\}\_\{r\+1\}\)\.\(33\)To define the local relative size weights used in the split consistency check, fix a levelr∈\{0,…,L−1\}r\\in\\\{0,\\dots,L\-1\\\}and letAk\(r\)A\_\{k\(r\)\}be the attribute on which every node at that level is split\. Forv∈ℒr\(𝒯\)v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\), write\{\(v,b\)\}b∈\{0,1\}\\\{\(v,b\)\\\}\_\{b\\in\\\{0,1\\\}\}for its two children\. Ifγ^r\+1↑\(v\)\>0\\hat\{\\gamma\}\_\{r\+1\}^\{\\uparrow\}\(v\)\>0, the local weights are obtained by restricting the level\-\(r\+1\)\(r\+1\)prior to the children ofvvand renormalizing:
γ^k\(r\),b\|𝒮v≜γ^r\+1\(v,b\)γ^r\+1↑\(v\),b∈\{0,1\}\.\\hat\{\\gamma\}\_\{k\(r\),b\\kern 0\.81949pt\|\\kern 0\.81949pt\\mathcal\{S\}\_\{v\}\}\\triangleq\\frac\{\\hat\{\\gamma\}\_\{r\+1\}\(v,b\)\}\{\\hat\{\\gamma\}\_\{r\+1\}^\{\\uparrow\}\(v\)\},\\qquad b\\in\\\{0,1\\\}\.\(34\)For an orderingπ\\pi, define the local split aggregate atv∈ℒr\(𝒯\)v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)by
T^h,agg\(𝒮v,Ak\(r\),π\)≜∑b∈\{0,1\}γ^k\(r\),b\|𝒮vT^h\(𝒮v,b,π𝒮v,b\)\.\\hat\{T\}\_\{h,\\mathrm\{agg\}\}\(\\mathcal\{S\}\_\{v\};A\_\{k\(r\)\};\\pi\)\\triangleq\\sum\_\{b\\in\\\{0,1\\\}\}\\hat\{\\gamma\}\_\{k\(r\),b\\kern 0\.81949pt\|\\kern 0\.81949pt\\mathcal\{S\}\_\{v\}\}\\,\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\}\_\{v,b\};\\pi\_\{\\mathcal\{S\}\_\{v,b\}\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\.\(35\)Ifγ^r\+1↑\(v\)=0\\hat\{\\gamma\}\_\{r\+1\}^\{\\uparrow\}\(v\)=0, we use the conventionT^h,agg\(𝒮v,Ak\(r\),π\)=T^h\(𝒮v,πv\)\\hat\{T\}\_\{h,\\mathrm\{agg\}\}\(\\mathcal\{S\}\_\{v\};A\_\{k\(r\)\};\\pi\)=\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\pi\_\{v\}\)\.444The choice does not affect the level\-\(r\+1\)\(r\+1\)aggregate becausevvreceives zero mass under the induced prior\. Setting its split aggregate equal to the direct estimate makes the associated split consistency check trivially satisfied\.Whenever no ordering is specified, the canonical ordering is used\. For two orderingsπ\\piandπ′\\pi^\{\\prime\}of the same set ofmmconstraints, letcay\(π,π′\)\\cay\(\\pi,\\pi^\{\\prime\}\)denote the transposition distance, defined as the minimum number of pairwise swaps required to transformπ\\piintoπ′\\pi^\{\\prime\}\. We defineσ≜π′∘π−1\\sigma\\triangleq\\pi^\{\\prime\}\\circ\\pi^\{\-1\}as the permutation of positions that carries the arrangementπ\\pito the arrangementπ′\\pi^\{\\prime\}\. Lettingc\(σ\)c\(\\sigma\)be its number of cycles \(counting fixed points\), the transposition distance satisfies Cayley’s identitycay\(π,π′\)=m−c\(σ\)\\cay\(\\pi,\\pi^\{\\prime\}\)=m\-c\(\\sigma\)\[[Diaconis, 1988](https://arxiv.org/html/2607.15277#bib.bib39), pp\. 112, 118\]\. Sincec\(σ\)≥1c\(\\sigma\)\\geq 1, this yields the diameter bound
cay\(π,π′\)≤m−1,\\cay\(\\pi,\\pi^\{\\prime\}\)\\leq m\-1,\(36\)with equality precisely whenσ\\sigmais anmm\-cycle\. Because a BCT splits on one shared attribute per level, every conditioning event𝒮v\\mathcal\{S\}\_\{v\}withv∈ℒr\(𝒯\)v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)is defined by the same set ofrrconstraints\. Thus,cay\(πv,π¯v\)\\cay\(\\pi\_\{v\},\\overline\{\\pi\}\_\{v\}\)does not depend onvv\. We denote this common value byκr\(𝒯,π\)\\kappa\_\{r\}\(\\mathcal\{T\},\\pi\), withκ0\(𝒯,π\)≜0\\kappa\_\{0\}\(\\mathcal\{T\},\\pi\)\\triangleq 0\. Applying \([36](https://arxiv.org/html/2607.15277#A4.E36)\) withm=rm=rthen gives
κr\(𝒯,π\)≤max\{r−1,0\}\.\\kappa\_\{r\}\(\\mathcal\{T\},\\pi\)\\leq\\max\\\{r\-1,\\,0\\\}\.\(37\)Further, we assume that the discrepancy metricdist\(∙,∙\)\\dist\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt,\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\)is jointly convex under finite mixtures\. That is, for any weightsλ1,…,λn≥0\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\\geq 0satisfying∑i=1nλi=1\\sum\_\{i=1\}^\{n\}\\lambda\_\{i\}=1, it holds that
dist\(∑iλiPi,∑iλiQi\)≤∑iλidist\(Pi,Qi\)\.\\dist\\left\(\{\\sum\}\_\{i\}\\lambda\_\{i\}P\_\{i\},\{\\sum\}\_\{i\}\\lambda\_\{i\}Q\_\{i\}\\right\)\\leq\\sum\_\{i\}\\lambda\_\{i\}\\,\\dist\(P\_\{i\},Q\_\{i\}\)\.\(38\)Finally, letℰh\\mathcal\{E\}\_\{h\}be the convex hull of the admissible values of the target functionalThT\_\{h\}, and define its diameter under the discrepancy metric as
Dh≜supU,V∈ℰhdist\(U,V\)\.D\_\{h\}\\triangleq\\sup\_\{U,V\\in\\mathcal\{E\}\_\{h\}\}\\dist\(U,V\)\.\(39\)We assume thatDhD\_\{h\}is finite\. For the normalized discrepancy metrics and target spaces used in the paper, we haveDh=1D\_\{h\}=1\.
###### Lemma 8\(Mixture sensitivity to weights\)\.
For estimatesT1,…,Tn∈ℰhT\_\{1\},\\dots,T\_\{n\}\\in\\mathcal\{E\}\_\{h\}and probability vectorsα,β∈Δn−1\\alpha,\\beta\\in\\Delta^\{n\-1\},
dist\(∑i=1nαiTi,∑i=1nβiTi\)≤DhTV\(α,β\)\.\\dist\\left\(\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\\kern 1\.00006ptT\_\{i\},\\sum\_\{i=1\}^\{n\}\\beta\_\{i\}\\kern 1\.00006ptT\_\{i\}\\right\)\\leq D\_\{h\}\\operatorname\{TV\}\(\\alpha,\\beta\)\.\(40\)
###### Proof\.
Letτ=TV\(α,β\)\\tau=\\operatorname\{TV\}\(\\alpha,\\beta\)andci=min\{αi,βi\}c\_\{i\}=\\min\\\{\\alpha\_\{i\},\\beta\_\{i\}\\\}\. The two mixtures share mass∑ici=1−τ\\sum\_\{i\}c\_\{i\}=1\-\\tau\. If0<τ<10<\\tau<1, we decompose both mixtures into their shared part with weight1−τ1\-\\tauand their respective residual parts with weightτ\\tau:
Tsh≜11−τ∑i=1nciTi,Tαres≜1τ∑i=1n\(αi−ci\)Ti,Tβres≜1τ∑i=1n\(βi−ci\)Ti\.T\_\{\\mathrm\{sh\}\}\\triangleq\\frac\{1\}\{1\-\\tau\}\\sum\_\{i=1\}^\{n\}c\_\{i\}\\kern 1\.00006ptT\_\{i\},\\qquad T\_\{\\alpha\}^\{\\mathrm\{res\}\}\\triangleq\\frac\{1\}\{\\tau\}\\sum\_\{i=1\}^\{n\}\(\\alpha\_\{i\}\-c\_\{i\}\)\\kern 1\.00006ptT\_\{i\},\\qquad T\_\{\\beta\}^\{\\mathrm\{res\}\}\\triangleq\\frac\{1\}\{\\tau\}\\sum\_\{i=1\}^\{n\}\(\\beta\_\{i\}\-c\_\{i\}\)\\kern 1\.00006ptT\_\{i\}\.\(41\)Joint convexity then gives
dist\(∑i=1nαiTi,∑i=1nβiTi\)≤\(1−τ\)dist\(Tsh,Tsh\)⏟=0\+τdist\(Tαres,Tβres\)≤τDh\.\\dist\\left\(\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\\kern 1\.00006ptT\_\{i\},\\sum\_\{i=1\}^\{n\}\\beta\_\{i\}\\kern 1\.00006ptT\_\{i\}\\right\)\\leq\(1\-\\tau\)\\,\{\\underbrace\{\\dist\(T\_\{\\mathrm\{sh\}\},T\_\{\\mathrm\{sh\}\}\)\}\_\{=\\;0\}\}\+\\tau\\dist\(T\_\{\\alpha\}^\{\\mathrm\{res\}\},T\_\{\\beta\}^\{\\mathrm\{res\}\}\)\\leq\\tau D\_\{h\}\.\(42\)The casesτ=0\\tau=0andτ=1\\tau=1follow directly\. ∎
###### Lemma 9\(Split consistency across levels\)\.
Fix0≤ℓ<ℓ′≤L0\\leq\\ell<\\ell^\{\\prime\}\\leq L\. If every internal node at levelsℓ,…,ℓ′−1\\ell,\\dots,\\ell^\{\\prime\}\-1, paired with its split attribute, isεSC\\varepsilon\_\{\\textnormal\{SC\}\}\-split consistent under the canonical ordering, then
dist\(T^h,agg\(ℓ\)\(𝒳,π¯\),T^h,agg\(ℓ′\)\(𝒳,π¯\)\)<\(ℓ′−ℓ\)εSC\+Dh∑j=ℓℓ′−1δj\(𝒯\),\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)<\(\\ell^\{\\prime\}\-\\ell\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\sum\_\{j=\\ell\}^\{\\ell^\{\\prime\}\-1\}\\delta\_\{j\}\(\\mathcal\{T\}\),\(43\)whereδr\(𝒯\)\\delta\_\{r\}\(\\mathcal\{T\}\)is the adjacent\-level prior\-refinement discrepancy\.
###### Proof\.
By the triangle inequality, we have
dist\(T^h,agg\(ℓ\)\(𝒳,π¯\),T^h,agg\(ℓ′\)\(𝒳,π¯\)\)≤∑j=ℓℓ′−1dist\(T^h,agg\(j\)\(𝒳,π¯\),T^h,agg\(j\+1\)\(𝒳,π¯\)\)\.\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)\\leq\\sum\_\{j=\\ell\}^\{\\ell^\{\\prime\}\-1\}\\dist\\left\(\\hat\{T\}^\{\(j\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(j\+1\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)\.\(44\)Thus, it suffices to bound the discrepancy between each pair of adjacent levels\. For fixedℓ≤j<ℓ′\\ell\\leq j<\\ell^\{\\prime\}, grouping the level\-\(j\+1\)\(j\+1\)nodes by their parents using the lifted relative size estimates \([34](https://arxiv.org/html/2607.15277#A4.E34)\) yields
T^h,agg\(j\+1\)\(𝒳,π¯\)\\displaystyle\\hat\{T\}^\{\(j\+1\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)=∑v∈ℒj\(𝒯\)γ^j\+1↑\(v\)∑b∈\{0,1\}γ^k\(j\),b\|𝒮vT^h\(𝒮v,b,π¯𝒮v,b\)\\displaystyle=\\sum\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\+1\}^\{\\uparrow\}\(v\)\\sum\_\{b\\in\\\{0,1\\\}\}\\hat\{\\gamma\}\_\{k\(j\),b\\kern 0\.81949pt\|\\kern 0\.81949pt\\mathcal\{S\}\_\{v\}\}\\,\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\}\_\{v,b\};\\overline\{\\pi\}\_\{\\mathcal\{S\}\_\{v,b\}\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\(45\)=∑v∈ℒj\(𝒯\)γ^j\+1↑\(v\)T^h,agg\(𝒮v,Ak\(j\),π¯\)\.\\displaystyle=\\sum\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\+1\}^\{\\uparrow\}\(v\)\\,\\hat\{T\}\_\{h,\\mathrm\{agg\}\}\(\\mathcal\{S\}\_\{v\};A\_\{k\(j\)\};\\overline\{\\pi\}\)\.\(46\)Using the shorthand notationQv≜T^h,agg\(𝒮v,Ak\(j\),π¯\)Q\_\{v\}\\triangleq\\hat\{T\}\_\{h,\\mathrm\{agg\}\}\(\\mathcal\{S\}\_\{v\};A\_\{k\(j\)\};\\overline\{\\pi\}\)and inserting the intermediate mixture∑vγ^j\(v\)Qv\\sum\_\{v\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006ptQ\_\{v\}gives
dist\(T^h,agg\(j\)\(𝒳,π¯\),T^h,agg\(j\+1\)\(𝒳,π¯\)\)≤dist\(T^h,agg\(j\)\(𝒳,π¯\),∑v∈ℒj\(𝒯\)γ^j\(v\)Qv\)\+dist\(∑v∈ℒj\(𝒯\)γ^j\(v\)Qv,∑v∈ℒj\(𝒯\)γ^j\+1↑\(v\)Qv\)\.\\begin\{split\}\\dist\\left\(\\hat\{T\}^\{\(j\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(j\+1\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)&\\leq\\dist\\left\(\\hat\{T\}^\{\(j\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006ptQ\_\{v\}\\right\)\\\\ &\+\\dist\\left\(\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006ptQ\_\{v\},\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\+1\}^\{\\uparrow\}\(v\)\\kern 1\.00006ptQ\_\{v\}\\right\)\.\\end\{split\}The first summand can be bounded by using joint convexity and local split consistency:
dist\(T^h,agg\(j\)\(𝒳,π¯\),∑v∈ℒj\(𝒯\)γ^j\(v\)Qv\)≤∑v∈ℒj\(𝒯\)γ^j\(v\)dist\(T^h\(𝒮v,π¯v\),Qv\)<εSC\.\\dist\\left\(\\hat\{T\}^\{\(j\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006ptQ\_\{v\}\\right\)\\leq\\sum\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006pt\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\overline\{\\pi\}\_\{v\}\),Q\_\{v\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\varepsilon\_\{\\textnormal\{SC\}\}\.\(47\)By[Lemma8](https://arxiv.org/html/2607.15277#Thmtheorem8), the second summand is bounded by
dist\(∑v∈ℒj\(𝒯\)γ^j\(v\)Qv,∑v∈ℒj\(𝒯\)γ^j\+1↑\(v\)Qv\)≤Dhδj\(𝒯\)\.\\dist\\left\(\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\}\(v\)\\kern 1\.00006ptQ\_\{v\},\{\\sum\}\_\{v\\in\\mathcal\{L\}\_\{j\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{j\+1\}^\{\\uparrow\}\(v\)\\kern 1\.00006ptQ\_\{v\}\\right\)\\leq D\_\{h\}\\kern 1\.00006pt\\delta\_\{j\}\(\\mathcal\{T\}\)\.\(48\)Combining these bounds with \([D\.1](https://arxiv.org/html/2607.15277#A4.Ex6)\) yields
dist\(T^h,agg\(j\)\(𝒳,π¯\),T^h,agg\(j\+1\)\(𝒳,π¯\)\)<εSC\+Dhδj\(𝒯\)\.\\dist\\left\(\\hat\{T\}^\{\(j\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(j\+1\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)<\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\delta\_\{j\}\(\\mathcal\{T\}\)\.\(49\)Summing overj=ℓ,…,ℓ′−1j=\\ell,\\dots,\\ell^\{\\prime\}\-1proves the claim\. ∎
###### Lemma 10\(Sufficiency of pairwise checks\)\.
Suppose[Assumption2](https://arxiv.org/html/2607.15277#Thmassumption2)holds and every \(pairwise\) check in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)is satisfied at toleranceε\\varepsilon\. Let𝒮=𝒮\(𝒥,𝐚𝒥\)\\mathcal\{S\}=\\mathcal\{S\}\(\\mathcal\{J\},\\bm\{a\}\_\{\\mathcal\{J\}\}\)be any conditioning event induced by𝒜\\mathcal\{A\}\. Then, for any two distinct orderingsπ≠π′\\pi\\neq\\pi^\{\\prime\}of its defining constraints,
dist\(T^h\(𝒮,π\),T^h\(𝒮,π′\)\)<cay\(π,π′\)ε≤\(\|𝒥\|−1\)ε\.\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi\),\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}<\\cay\(\\pi,\\pi^\{\\prime\}\)\\,\\varepsilon\\leq\(\|\\mathcal\{J\}\|\-1\)\\kern 1\.00006pt\\varepsilon\.\(50\)
###### Proof\.
Sinceπ≠π′\\pi\\neq\\pi^\{\\prime\}, the quantityq≜cay\(π,π′\)q\\triangleq\\cay\(\\pi,\\pi^\{\\prime\}\)satisfiesq≥1q\\geq 1\. Letσ≜π′∘π−1\\sigma\\triangleq\\pi^\{\\prime\}\\circ\\pi^\{\-1\}\. By definition of the transposition distance,σ\\sigmaadmits a decomposition intoqqtranspositions,σ=τq⋯τ1\\sigma=\\tau\_\{q\}\\cdots\\tau\_\{1\}\. Setπ\(t\)≜τt⋯τ1π\\pi^\{\(t\)\}\\triangleq\\tau\_\{t\}\\cdots\\tau\_\{1\}\\kern 1\.00006pt\\pifort∈\{0,…,q\}t\\in\\\{0,\\dots,q\\\}, so thatπ\(0\)=π\\pi^\{\(0\)\}=\\piandπ\(q\)=π′\\pi^\{\(q\)\}=\\pi^\{\\prime\}\. Consecutive orderingsπ\(t−1\)\\pi^\{\(t\-1\)\}andπ\(t\)\\pi^\{\(t\)\}differ by exchanging the positions of exactly two constraints with all remaining constraints held in place\. By[Assumption2](https://arxiv.org/html/2607.15277#Thmassumption2), the discrepancy induced by this swap equals the discrepancy of the corresponding two\-attribute check in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)\. Because every such check is satisfied at toleranceε\\varepsilon, we obtain, for everyt∈\{1,…,q\}t\\in\\\{1,\\dots,q\\\},
dist\(T^h\(𝒮,π\(t−1\)\),T^h\(𝒮,π\(t\)\)\)<ε\.\\dist\\\!\\left\(\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\};\\pi^\{\(t\-1\)\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\},\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\};\\pi^\{\(t\)\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\\right\)<\\varepsilon\.\(51\)The triangle inequality therefore gives
dist\(T^h\(𝒮,π\),T^h\(𝒮,π′\)\)\\displaystyle\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi\),\\hat\{T\}\_\{h\}\(\\mathcal\{S\};\\pi^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}≤∑t=1qdist\(T^h\(𝒮,π\(t−1\)\),T^h\(𝒮,π\(t\)\)\)\\displaystyle\\leq\\sum\_\{t=1\}^\{q\}\\dist\\\!\\left\(\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\};\\pi^\{\(t\-1\)\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\},\\hat\{T\}\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\mathcal\{S\};\\pi^\{\(t\)\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\\right\)\(52\)<qε=cay\(π,π′\)ε,\\displaystyle<q\\,\\varepsilon=\\cay\(\\pi,\\pi^\{\\prime\}\)\\,\\varepsilon,\(53\)which proves the first claim\. The second claim uses the fact that the transposition distance between two orderings of\|𝒥\|\|\\mathcal\{J\}\|constraints is at most\|𝒥\|−1\|\\mathcal\{J\}\|\-1\. ∎
### D\.2Within\-tree and cross\-tree consistency
The preceding lemma lifts local pairwise order consistency checks to arbitrary conditioning events\. Together with[Lemma9](https://arxiv.org/html/2607.15277#Thmtheorem9), it enables uniform comparisons of aggregate estimates across tree levels\. Specifically, the following theorem shows that the discrepancy between the aggregate estimates at any two levels is controlled by three quantities: the number of intervening splits, the prior\-refinement discrepancy accumulated across those splits, and the deviation of the path orderings at the two endpoint levels from their canonical orderings\.
###### Theorem 11\(Uniform consistency across tree levels\)\.
Let𝒜=\{A1,…,Ad\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{d\}\\\}, and suppose that
1. 1\.every\(𝒮,Ak\)∈𝒞SC\(𝒜\)\(\\mathcal\{S\},A\_\{k\}\)\\in\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\)isεSC\\varepsilon\_\{\\textnormal\{SC\}\}\-split consistent under the canonical ordering of the constraints defining𝒮\\mathcal\{S\};
2. 2\.every pairwise check in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)is satisfied at toleranceεOC\\varepsilon\_\{\\textnormal\{OC\}\}; and
3. 3\.[Assumptions2](https://arxiv.org/html/2607.15277#Thmassumption2)and[1](https://arxiv.org/html/2607.15277#Thmassumption1)hold\.
Let𝒯\\mathcal\{T\}be any binary conditioning tree of depthL≤dL\\leq d, induced by𝒜\\mathcal\{A\}, rooted at𝒳\\mathcal\{X\}, and equipped with a constraint orderingπ\\pi\. Then, for every0≤ℓ<ℓ′≤L0\\leq\\ell<\\ell^\{\\prime\}\\leq L,
dist\(T^h,agg\(ℓ\)\(𝒳,π\),T^h,agg\(ℓ′\)\(𝒳,π\)\)<\(ℓ′−ℓ\)εSC\+Dh∑j=ℓℓ′−1δj\(𝒯\)\+\[κℓ\(𝒯,π\)\+κℓ′\(𝒯,π\)\]εOC,\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\)\\right\)<\(\\ell^\{\\prime\}\-\\ell\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\sum\_\{j=\\ell\}^\{\\ell^\{\\prime\}\-1\}\\delta\_\{j\}\(\\mathcal\{T\}\)\+\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\\kappa\_\{\\ell\}\(\\mathcal\{T\},\\pi\)\+\\kappa\_\{\\ell^\{\\prime\}\}\(\\mathcal\{T\},\\pi\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\,\\varepsilon\_\{\\textnormal\{OC\}\},\(54\)whereκr\(𝒯,π\)\\kappa\_\{r\}\(\\mathcal\{T\},\\pi\)denotes the minimum number of pairwise swaps required to transform the attribute ordering at levelrrinduced byπ\\piinto the canonical order\.
###### Proof\.
We first bound the effect of replacing the path ordering by the canonical ordering at a fixed levelrr\. Since all pairwise checks in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\)are satisfied,[Assumptions2](https://arxiv.org/html/2607.15277#Thmassumption2)and[10](https://arxiv.org/html/2607.15277#Thmtheorem10)imply that, for everyv∈ℒr\(𝒯\)v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\),
dist\(T^h\(𝒮v,πv\),T^h\(𝒮v,π¯v\)\)≤cay\(πv,π¯v\)εOC\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\pi\_\{v\}\),\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\overline\{\\pi\}\_\{v\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\\leq\\cay\(\\pi\_\{v\},\\overline\{\\pi\}\_\{v\}\)\\,\\varepsilon\_\{\\textnormal\{OC\}\}\(55\)with strict inequality wheneverπv≠π¯v\\pi\_\{v\}\\neq\\overline\{\\pi\}\_\{v\}\. Whenπv=π¯v\\pi\_\{v\}=\\overline\{\\pi\}\_\{v\}, both sides of \([55](https://arxiv.org/html/2607.15277#A4.E55)\) are zero\. By[Assumption1](https://arxiv.org/html/2607.15277#Thmassumption1), the same level priorγ^r\\hat\{\\gamma\}\_\{r\}is used underπ\\piandπ¯\\overline\{\\pi\}\. Joint convexity ofdist\(∙,∙\)\\dist\(\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt,\\kern 1\.19995pt\\mathchoice\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\displaystyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\textstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptstyle\\bullet$\}\}\}\}\}\{\\mathbin\{\\vbox\{\\hbox\{\\scalebox\{\.48\}\{$\\scriptscriptstyle\\bullet$\}\}\}\}\}\\kern 1\.19995pt\), followed by \([55](https://arxiv.org/html/2607.15277#A4.E55)\), therefore gives
dist\(T^h,agg\(r\)\(𝒳,π\),T^h,agg\(r\)\(𝒳,π¯\)\)\\displaystyle\\dist\\left\(\\hat\{T\}^\{\(r\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(r\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)≤∑v∈ℒr\(𝒯\)γ^r\(v\)dist\(T^h\(𝒮v,πv\),T^h\(𝒮v,π¯v\)\)\\displaystyle\\leq\\sum\_\{v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{r\}\(v\)\\,\\dist\\\!\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\pi\_\{v\}\),\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\_\{v\};\\overline\{\\pi\}\_\{v\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\)$\}\}\}\(56\)≤εOC∑v∈ℒr\(𝒯\)γ^r\(v\)cay\(πv,π¯v\)\\displaystyle\\leq\\varepsilon\_\{\\textnormal\{OC\}\}\\sum\_\{v\\in\\mathcal\{L\}\_\{r\}\(\\mathcal\{T\}\)\}\\hat\{\\gamma\}\_\{r\}\(v\)\\,\\cay\(\\pi\_\{v\},\\overline\{\\pi\}\_\{v\}\)\(57\)=κr\(𝒯,π\)εOC\.\\displaystyle=\\kappa\_\{r\}\(\\mathcal\{T\},\\pi\)\\,\\varepsilon\_\{\\textnormal\{OC\}\}\.\(58\)Next,[Lemma9](https://arxiv.org/html/2607.15277#Thmtheorem9)bounds the discrepancy between the aggregates under canonical ordering:
dist\(T^h,agg\(ℓ\)\(𝒳,π¯\),T^h,agg\(ℓ′\)\(𝒳,π¯\)\)<\(ℓ′−ℓ\)εSC\+Dh∑j=ℓℓ′−1δj\(𝒯\)\.\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)<\(\\ell^\{\\prime\}\-\\ell\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\sum\_\{j=\\ell\}^\{\\ell^\{\\prime\}\-1\}\\delta\_\{j\}\(\\mathcal\{T\}\)\.\(59\)Finally, insert these two canonical order aggregates between the original aggregates\. By the triangle inequality, and the two bounds \([58](https://arxiv.org/html/2607.15277#A4.E58)\) and \([59](https://arxiv.org/html/2607.15277#A4.E59)\), we have
dist\(T^h,agg\(ℓ\)\(𝒳,π\),T^h,agg\(ℓ′\)\(𝒳,π\)\)\\displaystyle\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\)\\right\)≤dist\(T^h,agg\(ℓ\)\(𝒳,π\),T^h,agg\(ℓ\)\(𝒳,π¯\)\)\\displaystyle\\leq\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)\(60\)\+dist\(T^h,agg\(ℓ\)\(𝒳,π¯\),T^h,agg\(ℓ′\)\(𝒳,π¯\)\)\\displaystyle\+\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\)\\right\)\(61\)\+dist\(T^h,agg\(ℓ′\)\(𝒳,π¯\),T^h,agg\(ℓ′\)\(𝒳,π\)\)\\displaystyle\+\\dist\\left\(\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\overline\{\\pi\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\}\}\(\\mathcal\{X\};\\pi\)\\right\)\(62\)<\(ℓ′−ℓ\)εSC\+Dh∑j=ℓℓ′−1δj\(𝒯\)\+\[κℓ\(𝒯,π\)\+κℓ′\(𝒯,π\)\]εOC,\\displaystyle<\(\\ell^\{\\prime\}\-\\ell\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\sum\_\{j=\\ell\}^\{\\ell^\{\\prime\}\-1\}\\delta\_\{j\}\(\\mathcal\{T\}\)\+\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\\kappa\_\{\\ell\}\(\\mathcal\{T\},\\pi\)\+\\kappa\_\{\\ell^\{\\prime\}\}\(\\mathcal\{T\},\\pi\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\,\\varepsilon\_\{\\textnormal\{OC\}\},\(63\)which establishes the proof\. ∎
###### Corollary 12\(Cross\-tree consistency\)\.
Let𝒯\\mathcal\{T\}and𝒯′\\mathcal\{T\}^\{\\prime\}be any two binary conditioning trees induced by𝒜=\{A1,…,Ad\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{d\}\\\}, rooted at the same population𝒳\\mathcal\{X\}with depthsLLandL′L^\{\\prime\}, respectively\. Assume𝒯\\mathcal\{T\}and𝒯′\\mathcal\{T\}^\{\\prime\}are equipped with arbitrary constraint orderingsπ\\piandπ′\\pi^\{\\prime\}, respectively\. Under the conditions of[Theorem11](https://arxiv.org/html/2607.15277#Thmtheorem11), for every0≤ℓ≤L0\\leq\\ell\\leq Land0≤ℓ′≤L′0\\leq\\ell^\{\\prime\}\\leq L^\{\\prime\}with\(ℓ,ℓ′\)≠\(0,0\)\(\\ell,\\ell^\{\\prime\}\)\\neq\(0,0\), we have
dist\(T^h,agg,𝒯\(ℓ\)\(𝒳,π\),T^h,agg,𝒯′\(ℓ′\)\(𝒳,π′\)\)<\(ℓ\+ℓ′\)εSC\+Dh\[Δℓ\(𝒯\)\+Δℓ′\(𝒯′\)\]\+\[\(ℓ−1\)\+\+\(ℓ′−1\)\+\]εOC,\\begin\{split\}\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\\right\)&<\(\\ell\+\\ell^\{\\prime\}\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\\Delta\_\{\\ell\}\(\\mathcal\{T\}\)\+\\Delta\_\{\\ell^\{\\prime\}\}\(\\mathcal\{T\}^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\\\ &\+\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\(\\ell\-1\)\_\{\+\}\+\(\\ell^\{\\prime\}\-1\)\_\{\+\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{OC\}\},\\end\{split\}where\(x\)\+=max\{x,0\}\(x\)\_\{\+\}=\\max\\\{x,0\\\}andΔr\(𝒯\)≜∑j=0r−1δj\(𝒯\)\\Delta\_\{r\}\(\\mathcal\{T\}\)\\triangleq\\sum\_\{j=0\}^\{r\-1\}\\delta\_\{j\}\(\\mathcal\{T\}\)defines the cumulative prior\-refinement discrepancy through levelrr, withΔ0\(𝒯\)≜0\\Delta\_\{0\}\(\\mathcal\{T\}\)\\triangleq 0\.
###### Proof\.
The level\-zero aggregate is the direct estimate at the common root and is therefore independent of the tree:
T^h,agg,𝒯\(0\)\(𝒳,π\)=T^h\(𝒳\)=T^h,agg,𝒯′\(0\)\(𝒳,π′\)\.\\hat\{T\}^\{\(0\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\)=\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\)=\\hat\{T\}^\{\(0\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\.\(64\)For every1≤r≤L1\\leq r\\leq L, applying[Theorem11](https://arxiv.org/html/2607.15277#Thmtheorem11)within𝒯\\mathcal\{T\}to levels00andrrand then invoking \([37](https://arxiv.org/html/2607.15277#A4.E37)\) gives
dist\(T^h,agg,𝒯\(r\)\(𝒳,π\),T^h\(𝒳\)\)\\displaystyle\\dist\\left\(\\hat\{T\}^\{\(r\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\)\\right\)<rεSC\+DhΔr\(𝒯\)\+κr\(𝒯,π\)εOC\\displaystyle<r\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\kern 1\.00006pt\\Delta\_\{r\}\(\\mathcal\{T\}\)\+\\kappa\_\{r\}\(\\mathcal\{T\},\\pi\)\\,\\varepsilon\_\{\\textnormal\{OC\}\}\(65\)≤rεSC\+DhΔr\(𝒯\)\+\(r−1\)\+εOC,\\displaystyle\\leq r\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\kern 1\.00006pt\\Delta\_\{r\}\(\\mathcal\{T\}\)\+\(r\-1\)\_\{\+\}\\,\\varepsilon\_\{\\textnormal\{OC\}\},\(66\)The same bound holds within𝒯′\\mathcal\{T\}^\{\\prime\}for every1≤r≤L′1\\leq r\\leq L^\{\\prime\}\. Now recall that\(ℓ,ℓ′\)≠\(0,0\)\(\\ell,\\ell^\{\\prime\}\)\\neq\(0,0\)\. Using the direct root estimate as an intermediate point for the triangle inequality and applying \([66](https://arxiv.org/html/2607.15277#A4.E66)\) to each non\-zero level, we obtain
dist\(T^h,agg,𝒯\(ℓ\)\(𝒳,π\),T^h,agg,𝒯′\(ℓ′\)\(𝒳,π′\)\)\\displaystyle\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\\right\)\(67\)≤dist\(T^h,agg,𝒯\(ℓ\)\(𝒳,π\),T^h\(𝒳\)\)\+dist\(T^h\(𝒳\),T^h,agg,𝒯′\(ℓ′\)\(𝒳,π′\)\)\\displaystyle\\hskip\-68\.2866pt\\leq\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\)\\right\)\+\\dist\\left\(\\hat\{T\}\_\{h\}\(\\mathcal\{X\}\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\textnormal\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\\right\)\(68\)<\(ℓ\+ℓ′\)εSC\+Dh\[Δℓ\(𝒯\)\+Δℓ′\(𝒯′\)\]\+\[\(ℓ−1\)\+\+\(ℓ′−1\)\+\]εOC,\\displaystyle\\hskip\-68\.2866pt<\(\\ell\+\\ell^\{\\prime\}\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\kern 1\.00006pt\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\\Delta\_\{\\ell\}\(\\mathcal\{T\}\)\+\\Delta\_\{\\ell^\{\\prime\}\}\(\\mathcal\{T\}^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\+\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\(\\ell\-1\)\_\{\+\}\+\(\\ell^\{\\prime\}\-1\)\_\{\+\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{OC\}\},\(69\)where a term corresponding to a zero level vanishes by \([64](https://arxiv.org/html/2607.15277#A4.E64)\)\. This proves the claimed bound\. ∎
### D\.3Additional proof
###### Proof of[Proposition5](https://arxiv.org/html/2607.15277#Thmtheorem5)\.
Consider any two level aggregates, arising at levelsℓ\\ellandℓ′\\ell^\{\\prime\}, respectively, from BCTs induced by𝒜\\mathcal\{A\}and rooted at𝒳\\mathcal\{X\}\. Ifℓ=ℓ′=0\\ell=\\ell^\{\\prime\}=0, they coincide by \([64](https://arxiv.org/html/2607.15277#A4.E64)\)\. Otherwise,[Corollary12](https://arxiv.org/html/2607.15277#Thmtheorem12)gives
dist\(T^h,agg,𝒯\(ℓ\)\(𝒳,π\),T^h,agg,𝒯′\(ℓ′\)\(𝒳,π′\)\)\\displaystyle\\dist\\left\(\\hat\{T\}^\{\(\\ell\)\}\_\{h,\\mathrm\{agg\},\\mathcal\{T\}\}\(\\mathcal\{X\};\\pi\),\\hat\{T\}^\{\(\\ell^\{\\prime\}\)\}\_\{h,\\mathrm\{agg\},\\mathcal\{T\}^\{\\prime\}\}\(\\mathcal\{X\};\\pi^\{\\prime\}\)\\right\)<\(ℓ\+ℓ′\)εSC\+Dh\[Δℓ\(𝒯\)\+Δℓ′\(𝒯′\)\]\\displaystyle<\(\\ell\+\\ell^\{\\prime\}\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{SC\}\}\+D\_\{h\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\\Delta\_\{\\ell\}\(\\mathcal\{T\}\)\+\\Delta\_\{\\ell^\{\\prime\}\}\(\\mathcal\{T\}^\{\\prime\}\)\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\(70\)\+\[\(ℓ−1\)\+\+\(ℓ′−1\)\+\]εOC\\displaystyle\+\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\[$\}\}\}\(\\ell\-1\)\_\{\+\}\+\(\\ell^\{\\prime\}\-1\)\_\{\+\}\\vbox\{\\hbox\{\\scalebox\{1\.35\}\{$\]$\}\}\}\\varepsilon\_\{\\textnormal\{OC\}\}\(71\)≤2dεSC\+2\(d−1\)DhεPR\+2\(d−1\)εOC,\\displaystyle\\leq 2d\\,\\varepsilon\_\{\\textnormal\{SC\}\}\+2\(d\-1\)D\_\{h\}\\,\\varepsilon\_\{\\textnormal\{PR\}\}\+2\(d\-1\)\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{OC\}\},\(72\)where the second inequality usesℓ,ℓ′≤d\\ell,\\ell^\{\\prime\}\\leq d,δ0=0\\delta\_\{0\}=0, andΔr\(𝒯\)≤max\{r−1,0\}εPR\\Delta\_\{r\}\(\\mathcal\{T\}\)\\leq\\max\\\{r\-1,0\\\}\\kern 1\.00006pt\\varepsilon\_\{\\textnormal\{PR\}\}\. ∎
## Appendix EAdditional experimental results
This appendix reports additional analyses that probe the robustness of our experimental findings across prompting and implementation choices\.
### E\.1Partition\-then\-aggregate prompting
The aggregation experiments in[Section4](https://arxiv.org/html/2607.15277#S4)rely on an explicitly specified binary conditioning tree, where subgroup\-level estimates from a fixed partition are recombined via the law of total probability\. Their success raises a natural question: can the same benefit be recovered within a single query? We construct a prompt that instructs the model to first identify relevant subpopulations and then combine them into a population\-level estimate\. We call this strategy*partition\-then\-aggregate \(PTA\) prompting*\. Unlike explicit tree\-based aggregation, PTA prompting does not require an externally specified partition\. If successful, this would provide a lightweight and broadly applicable prompting strategy\. The exact prompt template is provided in[SectionH\.4](https://arxiv.org/html/2607.15277#A8.SS4)\.
−0\.25\-0\.250\.000\.000\.250\.250\.500\.500\.750\.75Average error gainGrok 4\.20Gemini 3\.1 ProSonnet 4\.6Opus 4\.7GPT\-4o miniGPT\-5\.410020k40k60k80kThresholds \(USD\)GPT\-5\.4GPT\-4o miniOpus 4\.7Sonnet 4\.6Gemini 3\.1 ProGrok 4\.20
\(a\)Average error gain\.Average relative error gain of PTA prompting over direct prompting across the five income thresholds\. Positive values indicate that PTA prompting reduces absolute error relative to direct prompting\.\(b\)Win matrix\.Each tile corresponds to one model–threshold pair\. Green indicates that PTA prompting yields lower absolute error than direct prompting; red indicates the complement\.
Figure 10:Benefit of PTA prompting on ACS income estimation\.We compare direct prompting with PTA prompting, which asks the model to first reason over relevant subpopulations before returning an aggregate prediction, while leaving the subpopulations implicit\. Unlike explicit tree\-based aggregation, the decomposition is not fixed in advance but chosen implicitly by the model\. For each model and income thresholdτ∈\{100,20k,40k,60k,80k\}\\tau\\in\\\{100,20\\mathrm\{k\},40\\mathrm\{k\},60\\mathrm\{k\},80\\mathrm\{k\}\\\}USD, we evaluate the absolute error between the predicted population\-level probabilityℙ\(Y\>τ\|X∈𝒳\)\\mathbb\{P\}\(Y\>\\tau\\kern 0\.92505pt\|\\kern 0\.92505ptX\\in\\mathcal\{X\}\)and the ACS survey\-based ground truth\.To evaluate this idea, we compare PTA prompting against a direct prompting baseline, in which the model is asked to estimate the aggregate target quantity in a single step without any reference to subpopulations\. On the ACS income task,[Figure10\(a\)](https://arxiv.org/html/2607.15277#A5.F10.sf1)shows that PTA prompting often improves over direct prompting when averaged across the five income thresholds\. However, the gains are model\-dependent and less systematic than those obtained with explicit tree\-based aggregation\. PTA prompting therefore recovers part of the benefit of decomposition through a lightweight implicit strategy, but it is not uniformly beneficial across models\. A more detailed analysis of when model\-chosen decompositions help is left for future work\.
### E\.2Self\-consistency across problem domains
This subsection reports the additional self\-consistency results\. We first vary the prediction target within the ACS setting and then move to opinion distributions from the World Values Survey\.
#### ACS prediction tasks\.
First, we evaluate self\-consistency on the ACS setting discussed in the main body\. The ACS income prediction task corresponds to a widely used testbed for evaluating alignment and fairness of predictive models\[[Ding et al\., 2021](https://arxiv.org/html/2607.15277#bib.bib22),[Cruz et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib16)\]\. We take𝒜\\mathcal\{A\}to be the demographic attributes underlying the tree in[Section3](https://arxiv.org/html/2607.15277#S3)and, for every conditioning event induced by𝒜\\mathcal\{A\}, elicit the model’s estimated binned income distribution\. We then apply the self\-consistency checks from[Section5](https://arxiv.org/html/2607.15277#S5)\. The two columns in[Table2\(a\)](https://arxiv.org/html/2607.15277#A5.T2.st1)report the two complementary benchmark scores\. The first is the split consistency scoreSCε\(𝒜\)\\textnormal\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)\. For every admissible pair of a conditioning event and a next split attribute in𝒞SC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{SC\}\}\(\\mathcal\{A\}\), we compare the direct estimate to the prior\-weighted aggregate of its two refinements and report the fraction of checks satisfied at toleranceε=0\.02\\varepsilon=0\.02\. The second is the order consistency scoreOCε\(𝒜\)\\textnormal\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)\. For every two\-attribute conditioning event in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\textnormal\{OC\}\}\(\\mathcal\{A\}\), we elicit direct estimates under both orderings of the attribute constraints and report the fraction of pairs whose predictions agree within toleranceε=0\.02\\varepsilon=0\.02\. Across frontier models, we observe widespread violations of both criteria\. These discrepancies point to an important weakness of model\-induced estimates and help explain the varying alignment scores of reconstructed estimates across levels\.
We next hold the population structure fixed while changing the prediction target\. In the binned ACS commute time task reported in[Table2\(b\)](https://arxiv.org/html/2607.15277#A5.T2.st2), we use the same conditioning attributes and subgroup priors as in the income experiment, but ask models to estimate a different target distribution\. The resulting split and order consistency scores differ substantially from those obtained for income, showing that self\-consistency is not solely a property of the model or conditioning tree, but can also vary across prediction tasks\.
\(a\)ACS income prediction\.The target variableYYis the binned income distribution\.
\(b\)ACS commute time prediction\.The target variableYYis the binned commute time distribution\.
Table 2:Self\-consistency in ACS prediction tasks\.We evaluate self\-consistency for a fixed tree and two different prediction tasks\. The attribute set𝒜\\mathcal\{A\}consists of age and employment status as used in[Figure14](https://arxiv.org/html/2607.15277#A6.F14)\. For each task, we report the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02\. Higher values indicate stronger self\-consistency and are shown in lighter red tones\. We observe widespread violations of self\-consistency across frontier models\.
#### Global opinion modeling\.
The split and order consistency scores cover all conditioning trees induced by a fixed attribute set𝒜\\mathcal\{A\}\(see[Proposition5](https://arxiv.org/html/2607.15277#Thmtheorem5)\)\. A consistent conditional estimator, however, must satisfy these identities not only across trees, but for every attribute set and every problem domain\. To probe this problem axis, we construct binary conditioning trees from the World Values Survey \(WVS\), extending our evaluation to opinion distributions across countries beyond the United States\. Building on the GlobalOpinionQA dataset\[[Durmus et al\., 2024](https://arxiv.org/html/2607.15277#bib.bib20)\], we use demographic attributes as conditioning variables and examine, for each country, how subpopulation\-level estimates aggregate to the corresponding country\-level estimate\. Each survey question thereby defines a distinct prediction problem on which the consistency checks are instantiated separately\.
Table 3:Self\-consistency in global opinion prediction\.For five survey questions sampled from the WVS\(Qa,…,Qe\)\(Q\_\{a\},\\dots,Q\_\{e\}\)and two countries, we report the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02, where𝒜\\mathcal\{A\}consists of age \(younger than 45 years old vs\. at least 45 years old\) and income splits \(low vs\. high\)\. Each survey question defines a distinct prediction problem\. Higher values indicate stronger self\-consistency and are shown in lighter red tones\. Across models, countries, and questions, current models fail to satisfy the self\-consistency checks reliably\.We run our evaluation on data from Canada and Indonesia, the two countries with the largest sample sizes\. For each country, we construct a BCT of depth two using age and income splits\. We then elicit categorical response distributions at every node for five selected survey questions\. The full question wording and answer scales are reported in[AppendixJ](https://arxiv.org/html/2607.15277#A10)\. Consistency is assessed using the reference\-free checks from[Section5](https://arxiv.org/html/2607.15277#S5)\. For each question, this includes the split and order consistency checks\. Results for toleranceε=0\.02\\varepsilon=0\.02are reported in[Table3](https://arxiv.org/html/2607.15277#A5.T3)\.
The WVS results show that the self\-consistency failures observed on ACS are not specific to income prediction or to the U\.S\. census setting\. Across both countries, models satisfy only a fraction of the split and order consistency checks\. No model is uniformly consistent across questions\. Even the strongest models exhibit scores that vary substantially across survey questions and countries\. Thus, the WVS provides complementary evidence that current models do not elicit faithful probability estimates\.
### E\.3Self\-consistency in decision models
We extend the evaluation in[Figure4](https://arxiv.org/html/2607.15277#S5.F4)to decision models, which are trained specifically for decision tasks and calibrated probability estimation\. We retain the thresholded ACS income prediction task withτ=40k\\tau=40\\mathrm\{k\}USD, demographic attribute set𝒜\\mathcal\{A\}, and consistency checks\. Decision models require a predefined structured output format, which we provide in[SectionsH\.8](https://arxiv.org/html/2607.15277#A8.SS8)and[H\.9](https://arxiv.org/html/2607.15277#A8.SS9)together with the prompt and context\. We evaluate split and order consistency at toleranceε=0\.02\\varepsilon=0\.02\.[Figure11](https://arxiv.org/html/2607.15277#A5.F11)shows that, despite their explicit focus on reliable probability estimation, these models violate basic probabilistic consistency requirements\. Structured outputs and calibration\-oriented training thus leave a fundamental reliability gap\.
Jev 1\.13Jev 1\.14Liquid D1Liquid D1 UltraGPT\-6 Luna DecisionsMicrosoft Decision 1Drex 1\.5Mercury Decide0\.000\.000\.250\.250\.500\.500\.750\.751\.001\.00Consistency scoreSplit consistencyOrder consistencyFigure 11:Decision models violate probabilistic self\-consistency\.We report the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02for the thresholded ACS income prediction task withτ=40k\\tau=40\\mathrm\{k\}USD, following the experimental setting of[Figure4](https://arxiv.org/html/2607.15277#S5.F4)\. Despite training specifically for decision tasks and calibrated probability estimation, these models fail basic consistency checks across their own predictions\.
### E\.4Level\-wise self\-consistency
Table 4:Level\-wise root consistency\.For each levelℓ\\ellof a depth\-four tree, we reconstruct the aggregate income distribution from the corresponding level\-ℓ\\ellpartition and compare it to the model’s direct root estimate\. Entries report the normalized Wasserstein\-11distance\. Lower values indicate stronger agreement with the root estimate and are highlighted using darker green tones\.To assess how self\-consistency varies with the granularity of the partition,[Table4](https://arxiv.org/html/2607.15277#A5.T4)reports a level\-wise root consistency evaluation\. For this experiment, we use the first four levels of the depth\-six binary conditioning tree described in[SectionF\.3](https://arxiv.org/html/2607.15277#A6.SS3)\. For each levelℓ\\ell, we reconstruct the aggregate income distribution from the partition induced at that level and compare it to the model’s direct root estimate using the normalized Wasserstein distance𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}\. The table reports these discrepancies directly, so lower values indicate stronger agreement with the root estimate\.
### E\.5Prompt sensitivity
0\.00\.00\.20\.20\.40\.40\.60\.60\.80\.81\.01\.0True probability0\.00\.00\.20\.20\.40\.40\.60\.60\.80\.81\.01\.0Predicted probability0\.00\.00\.20\.20\.40\.40\.60\.60\.80\.81\.01\.0True probability0\.00\.00\.20\.20\.40\.40\.60\.60\.80\.81\.01\.0Predicted probabilityDirectAggregated
\(a\)Sociodemographic prompting\.The target subpopulation is described in the third person using structured attributes\.\(b\)Persona prompting\.The model is instructed to role\-play as a member of the target subpopulation in the first person\.
Figure 12:Prompt sensitivity of probability elicitation\.Results are shown for GPT\-5\.4 without reasoning, using ground\-truth priors and 100 repeated LLM samples\. For multiple income thresholds, model estimates are plotted against the corresponding survey\-derived probabilities\. Points closer to the diagonal indicate stronger ground\-truth alignment\. Direct estimates are obtained from population\-level prompts, whereas reconstructed estimates are obtained by aggregating subgroup\-level estimates via the law of total probability using ground\-truth priors\. The population is decomposed according to employment status; see theESRattribute in[AppendixI](https://arxiv.org/html/2607.15277#A9)\. Error bars correspond to 90 % confidence intervals computed by bootstrapping with 1000 samples\. Error bars are barely visible as the intervals are very narrow\.We examine prompt sensitivity along two complementary dimensions: ground\-truth alignment and reference\-free self\-consistency\. In the default*sociodemographic*formulation, the target subpopulation is described in the third person through a structured list of attributes; see the prompt template in[SectionH\.2](https://arxiv.org/html/2607.15277#A8.SS2)\. In the*persona prompting*formulation, the same information is instead presented as a first\-person role\-playing instruction; see[SectionH\.3](https://arxiv.org/html/2607.15277#A8.SS3)\. Both formulations elicit the same verbalized probability estimate for the same event and differ only in how the conditioning information is expressed\.
We first assess ground\-truth alignment in the thresholded ACS income task\. Across the two prompt formulations, we keep the model, sampling temperature, number of repeated generations, outcome event, income thresholds, and aggregation weights fixed\. The resulting comparisons with the survey\-derived probabilities are shown in[Figure12](https://arxiv.org/html/2607.15277#A5.F12)\. Under both formulations, estimates reconstructed through the law of total probability lie closer to the diagonal than direct population\-level estimates for most thresholds\. The benefit of subgroup decomposition is therefore not specific to a particular formulation of the conditioning context\. However, persona prompting systematically overestimates the ground\-truth probabilities, whereas sociodemographic prompting produces more closely aligned estimates\. We therefore use sociodemographic prompting as the default throughout our experiments\.
We next examine the complementary, reference\-free question of whether the elicited estimates are statistically self\-consistent\. For a subset of models, we repeat the WVS evaluation from[Table3](https://arxiv.org/html/2607.15277#A5.T3), changing only the prompting method\. Specifically, we replace the default third\-person sociodemographic template with the first\-person persona template\.[Table5](https://arxiv.org/html/2607.15277#A5.T5)reports the fraction of self\-consistency checks satisfied by each model, revealing that the effect of persona prompting varies across models\. It lowers self\-consistency for some models while improving it for others\. Averaged over the six models evaluated under both prompt formulations, the split consistency rate decreases from0\.4750\.475under sociodemographic prompting to0\.4030\.403under persona prompting\. The order consistency rate increases slightly from0\.5870\.587to0\.5920\.592\. Thus, although prompt formulation affects individual consistency rates, substantial violations persist under both interfaces, leaving our central qualitative finding unchanged\.
Table 5:Robustness to prompt formulation\.Self\-consistency results under first\-person persona prompting, using the same experimental setting, notation, and evaluation criteria as[Table3](https://arxiv.org/html/2607.15277#A5.T3)\. Higher values indicate stronger self\-consistency\. The effects vary across models, and substantial violations persist\.
### E\.6Sensitivity to split verbalization
We examine whether the aggregation gains reported in[Section4](https://arxiv.org/html/2607.15277#S4)depend on how the conditioning\-attribute splits are expressed in natural language\. To isolate this effect, we repeat the thresholded ACS income experiment while varying only the textual descriptions of the first three splits\. The underlying partitions remain unchanged\. We compare the default verbalization, which explicitly enumerates the attribute values assigned to each branch, with a simplified verbalization that expresses the same sets using compact intervals or semantically meaningful category descriptions\. Specifically, at level one, theAGEPbranches are described using age ranges\. For example, the explicit enumeration “31 years old or 32 years old or …” is replaced by “31 to 68 years old\.” At level two, theCOWvalues\{1,…,7\}\\\{1,\\ldots,7\\\}are summarized as “currently employed or self\-employed,” whereas values\{8,9,n/a\}\\\{8,9,\\texttt\{n/a\}\\\}are summarized as “not currently in paid employment\.” These groupings preserve the ACS category mapping reported in[Table7](https://arxiv.org/html/2607.15277#A9.T7)\. Finally, at level three, theWKHPsplit is expressed using compact intervals, such as “1 to 39 hours per week,” instead of explicitly listing every constituent value\.
0123Level−\-25%0%25%50%75%Aggregation gain0123Level0%20%40%60%80%Aggregation gainτ=1k\\tau=1\\mathrm\{k\}USDτ=10k\\tau=10\\mathrm\{k\}USDτ=20k\\tau=20\\mathrm\{k\}USDτ=40k\\tau=40\\mathrm\{k\}USDτ=60k\\tau=60\\mathrm\{k\}USDτ=1k\\tau=1\\mathrm\{k\}USDτ=10k\\tau=10\\mathrm\{k\}USDτ=20k\\tau=20\\mathrm\{k\}USDτ=40k\\tau=40\\mathrm\{k\}USDτ=60k\\tau=60\\mathrm\{k\}USD
\(a\)Default verbalization\.The attribute values defining each branch are enumerated explicitly\.\(b\)Simplified verbalization\.The same partitions are described using compact intervals and natural\-language category summaries\.
Figure 13:Sensitivity of aggregation gains to split verbalization\.We report the aggregation gain across the first three levels of the thresholded ACS income tree under the default and simplified verbalization schemes\. All experimental choices other than the subgroup descriptions are held fixed\. Positive values indicate that estimates reconstructed from subgroup predictions are better aligned with the survey\-derived reference than the direct population\-level estimate at level 0\. Bars correspond to different income thresholds, and the orange curve shows their average\. Error bars denote 90 % bootstrap confidence intervals\. Aggregation gains persist under both schemes and are more uniformly positive under the simplified descriptions\.As shown in[Figure13](https://arxiv.org/html/2607.15277#A5.F13), changing the split descriptions affects the magnitude of the resulting aggregation gains\. Nevertheless, the central qualitative pattern remains unchanged: estimates reconstructed from finer\-grained subgroup predictions generally improve upon direct population\-level prompting under both verbalization schemes\. These results show that the macro fallacy is not an artifact of the split verbalization method\.
### E\.7Robustness to reasoning effort
We assess robustness to reasoning effort by repeating the self\-consistency evaluation from[Table3](https://arxiv.org/html/2607.15277#A5.T3)under the same experimental setup, but instructing the models to use a “low” reasoning budget\. We compare these results with the corresponding runs without reasoning while keeping all other evaluation choices fixed\. Because explicit reasoning controls are not available for every model, the analysis is restricted to models for which both settings can be evaluated\.[Table6](https://arxiv.org/html/2607.15277#A5.T6)reports the fraction of self\-consistency checks satisfied by each model\.
Enabling reasoning affects individual model performance, but the effects vary considerably across models\. Averaged across the evaluated models, the split consistency rate increases only modestly, from0\.4980\.498without reasoning to0\.5120\.512with a low reasoning budget\. The order consistency rate exhibits a similarly small increase, from0\.6340\.634to0\.6520\.652\. Thus, additional reasoning does not reliably eliminate the observed statistical consistency violations\. Our central qualitative finding is therefore robust to enabling low\-effort reasoning\.
Table 6:Robustness to reasoning effort\.Self\-consistency results obtained with a low reasoning budget, using the same experimental setting, notation, and evaluation criteria as[Table3](https://arxiv.org/html/2607.15277#A5.T3)\. Higher values indicate stronger self\-consistency\. Additional reasoning does not reliably eliminate self\-consistency violations\.
## Appendix FExperiment details
### F\.1Binary conditioning tree
Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Income0\.00\.00\.20\.20\.40\.4ℙ\\mathbb\{P\}Income0\.00\.00\.50\.5ℙ\\mathbb\{P\}unemployedIncome0\.00\.00\.20\.20\.40\.4ℙ\\mathbb\{P\}employedyoung & oldIncome0\.00\.00\.20\.20\.40\.4ℙ\\mathbb\{P\}Income0\.00\.00\.20\.20\.40\.4ℙ\\mathbb\{P\}unemployedIncome0\.00\.00\.20\.20\.40\.4ℙ\\mathbb\{P\}employedworking ageACSLLMIncome bin\[−11\.5k,1\)\[\-11\.5\\mathrm\{k\},1\)USD\[1,25k\)\[1,25\\mathrm\{k\}\)USD\[25k,60k\)\[25\\mathrm\{k\},60\\mathrm\{k\}\)USD\[60k,1849k\]\[60\\mathrm\{k\},1849\\mathrm\{k\}\]USD
Figure 14:Ground\-truth and LLM\-elicited income distributions across partitions of the population\.We construct a binary conditioning tree from the 2024 ACS population by recursively slicing the population with sociodemographic attributes\. The first split criterion corresponds to age \(AGEP\) and the second to class of worker \(COW\)\. Each node reports the distribution of yearly income over four bins\. For each bin, the solid bar shows the survey\-based ground\-truth distribution, while the hatched bar shows the LLM\-estimated conditional distribution\.0102030405060708090Age00200002000040000400006000060000Mean Income9n/a86314257Class of worker002000020000400004000060000600008000080000100000100000Mean Income
\(a\)Age attributeAGEP\.\(b\)Class\-of\-worker attributeCOW\.
Figure 15:Mean income across attribute\-induced subgroups\.We partition the ACS population by all attainable values of a single attribute and report the survey\-weighted mean yearly income in USD for each resulting subgroup\. For the age attribute in[Figure15\(a\)](https://arxiv.org/html/2607.15277#A6.F15.sf1), subgroup values are displayed in their natural order, and the binary split is defined by thresholding the corresponding subgroup mean incomes \(dashed line\)\. For the categorical class\-of\-worker attribute in[Figure15\(b\)](https://arxiv.org/html/2607.15277#A6.F15.sf2), subgroup values are first sorted by mean income before defining the split\. Both rules yield splits that separate lower\-income from higher\-income attribute values while keeping the two branches approximately balanced in survey\-weighted probability mass\.For our running example in[Section3](https://arxiv.org/html/2607.15277#S3), we construct the binary conditioning tree shown in[Figure14](https://arxiv.org/html/2607.15277#A6.F14)using demographic attributes from the 2024 American Community Survey \(ACS\) 1\-Year Public Use Microdata Sample\[[U\.S\. Census Bureau, 2024](https://arxiv.org/html/2607.15277#bib.bib24)\]\. The root node corresponds to the U\.S\. population, which we recursively refine using two binary splits\. To select these splits, we consider the following candidate attributes:AGEP\(age\),CIT\(citizenship\),COW\(class of worker\),ESR\(employment status\),MAR\(marital status\),OCCP\(occupation\),SCHL\(educational attainment\),SEX\(sex\), andWKHP\(usual hours worked per week\)\. For each candidate attribute, we first partition the population by all attainable attribute values and compute the mean yearly income within each resulting subgroup, as exemplified in[Figure15](https://arxiv.org/html/2607.15277#A6.F15)\. We then group the attribute values into a lower\-income and a higher\-income branch, choosing the grouping so that the two branches have approximately equal survey\-weighted probability mass\. Finally, we qualitatively select the two attributes whose induced binary splits yield particularly distinctive income distributions across the two branches\. This procedure yields the two splits used in our main ACS experiments\. The first split is induced by the ACS age attributeAGEP\. Respondents younger than3131or older than6868are grouped into the “young & old” subgroup, while respondents aged3131to6868are grouped into the “working age” subgroup\. The second split uses the class\-of\-worker attributeCOW\. Values11–77are labeled as “employed,” whereas values88,99, and missing values are mapped to “unemployed\.” The corresponding ACS attribute descriptions are reported in[Table7](https://arxiv.org/html/2607.15277#A9.T7)\.
At each node in[Figure14](https://arxiv.org/html/2607.15277#A6.F14), we report the survey\-weighted distribution of yearly income in USD\. For the four\-bin income setting, the bin edges are chosen so that the root distribution is approximately uniform, and the same binning is then used for all subpopulations in the tree\. Since yearly income is unavailable for respondents below age1515, we set their income to00USD in our experiments, reflecting the assumption that individuals in this age group have no reported labor income\.
### F\.2Prior elicitation
To obtain fully model\-induced aggregate quantities, both the subgroup\-conditional estimates and the subgroup priors must be elicited from the LLM\. In this subsection, we describe the level\-wise procedure used to estimate these priors\. For each tree levelℓ\\ellwith induced partition𝒫ℓ\\mathcal\{P\}\_\{\\ell\}, we prompt the modeln=50n=50times to estimate the probability mass that a randomly sampled individual from the base population belongs to each subgroup𝒮∈𝒫ℓ\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\. The prompts use the template in[SectionH\.1](https://arxiv.org/html/2607.15277#A8.SS1), which lists the subgroups in the partition and asks the model to return a normalized probability distribution over them\. Although the prompt instructs the model to output probabilities that sum to one, the raw responses may contain small normalization errors\. We therefore post\-process each response before averaging\. The LLM\-estimated prior at levelℓ\\ellis then obtained by averaging the renormalized estimates across repetitions\.
### F\.3Alignment error
The alignment analysis in[Figure2](https://arxiv.org/html/2607.15277#S4.F2)is based on a binary conditioning tree over ACS attributes\. The tree is defined by the following sequence of splits:
- Level 1:AGEPis split into\{0, …, 30, 69, …, 96, n/a\}and\{31, …, 68\}\.
- Level 2:COWis split into\{1, …, 7\}and\{8, 9, n/a\}\.
- Level 3:WKHPis split into\{1, …, 39, n/a\}and\{40, …, 99\}\.
- Level 4:OCCPis split into\{2060, 2300, 2740, 3401, 3421, 3422, 3424, 3603, 3605, 3620, 3630, 3640, 3645, 3647, 3648, 3649, 3940, 3960, 2350, 2440, 2545, 2633, 2723, 3430, 3601, 3602, 3646, 3946, 4010, 4020, 4030, 4040, 4055, 4110, 4120, 4130, 4140, 4150, 4160, 4220, 4230, 4251, 4255, 4350, 4420, 4435, 4461, 4500, 4510, 4521, 4522, 4525, 4530, 4540, 4600, 4621, 4622, 4640, 4655, 4720, 4740, 4760, 4900, 4940, 5020, 5160, 5260, 5300, 5310, 5320, 5400, 5510, 5610, 5810, 5820, 5850, 5860, 5900, 6040, 6050, 6120, 6410, 6600, 7260, 7510, 7610, 7800, 7810, 7830, 7840, 7850, 8256, 8300, 8310, 8320, 8335, 8350, 8365, 8465, 8530, 8540, 8710, 8800, 8850, 8910, 8950, 9110, 9142, 9350, 9365, 9415, 9600, 9610, 9620, 9630, 9640, 9645, 9720, 9920, n/a\}and\{10, 20, 40, 51, 52, 60, 101, 102, 110, 120, 135, 136, 137, 140, 150, 160, 205, 220, 230, 300, 310, 335, 340, 350, 360, 410, 420, 425, 440, 500, 510, 520, 530, 540, 565, 600, 630, 640, 650, 700, 705, 710, 725, 726, 735, 750, 800, 810, 820, 830, 845, 850, 860, 900, 910, 930, 940, 960, 1005, 1006, 1007, 1010, 1021, 1022, 1031, 1032, 1050, 1065, 1105, 1106, 1108, 1200, 1220, 1240, 1305, 1306, 1310, 1320, 1340, 1350, 1360, 1400, 1410, 1420, 1430, 1440, 1450, 1460, 1520, 1530, 1541, 1545, 1551, 1555, 1560, 1600, 1610, 1640, 1650, 1700, 1710, 1720, 1745, 1750, 1760, 1800, 1821, 1822, 1825, 1840, 1860, 1900, 1910, 1920, 1935, 1970, 1980, 2001, 2002, 2003, 2004, 2005, 2006, 2011, 2012, 2013, 2014, 2015, 2016, 2025, 2040, 2050, 2100, 2105, 2145, 2170, 2180, 2205, 2310, 2320, 2330, 2360, 2400, 2435, 2555, 2600, 2631, 2632, 2634, 2635, 2636, 2640, 2700, 2710, 2721, 2722, 2751, 2752, 2755, 2770, 2805, 2810, 2825, 2830, 2840, 2850, 2861, 2862, 2865, 2905, 2910, 2920, 3000, 3010, 3030, 3040, 3050, 3090, 3100, 3110, 3120, 3140, 3150, 3160, 3200, 3210, 3220, 3230, 3245, 3250, 3255, 3256, 3258, 3261, 3270, 3300, 3310, 3321, 3322, 3323, 3324, 3330, 3402, 3423, 3500, 3515, 3520, 3545, 3550, 3610, 3655, 3700, 3710, 3720, 3725, 3740, 3750, 3801, 3802, 3820, 3840, 3870, 3900, 3910, 3930, 3945, 4000, 4200, 4210, 4240, 4252, 4330, 4340, 4400, 4465, 4700, 4710, 4750, 4800, 4810, 4820, 4830, 4840, 4850, 4920, 4930, 4950, 4965, 5000, 5010, 5040, 5100, 5110, 5120, 5140, 5150, 5165, 5220, 5230, 5240, 5250, 5330, 5340, 5350, 5360, 5410, 5420, 5500, 5521, 5522, 5530, 5540, 5550, 5560, 5600, 5630, 5710, 5720, 5730, 5740, 5840, 5910, 5920, 5940, 6005, 6010, 6115, 6130, 6200, 6210, 6220, 6230, 6240, 6250, 6260, 6305, 6330, 6355, 6360, 6400, 6441, 6442, 6460, 6515, 6520, 6530, 6540, 6660, 6700, 6710, 6720, 6730, 6740, 6765, 6800, 6825, 6835, 6850, 6950, 7000, 7010, 7020, 7030, 7040, 7100, 7120, 7130, 7140, 7150, 7160, 7200, 7210, 7220, 7240, 7300, 7315, 7320, 7330, 7340, 7350, 7360, 7410, 7420, 7430, 7540, 7560, 7640, 7700, 7720, 7730, 7740, 7750, 7855, 7905, 7925, 7950, 8000, 8025, 8030, 8040, 8100, 8130, 8140, 8225, 8250, 8255, 8450, 8500, 8510, 8555, 8600, 8610, 8620, 8630, 8640, 8650, 8720, 8730, 8740, 8750, 8760, 8810, 8830, 8920, 8930, 8940, 8990, 9005, 9030, 9040, 9050, 9121, 9122, 9130, 9141, 9150, 9210, 9240, 9265, 9300, 9310, 9410, 9430, 9510, 9570, 9650, 9760, 9800, 9810, 9825, 9830\}\.
- Level 5:ESRis split into\{1, 2, 4, 5\}and\{3, 6, n/a\}\.
- Level 6:MARis split into\{1, 3\}and\{2, 4, 5, n/a\}\.
The ACS category mapping is reported in[Table7](https://arxiv.org/html/2607.15277#A9.T7)\. For each levelℓ∈\{0,1,…,6\}\\ell\\in\\\{0,1,\\dots,6\\\}, we compute the alignment errorAErr\(ℓ\)\\mathrm\{AErr\}\(\\ell\)of the aggregate estimate obtained from the nodes at that level\. To compare errors across levels, we normalize by the root\-level error and report1−AErr\(ℓ\)/AErr\(0\)1\-\\text\{AErr\}\(\\ell\)/\\text\{AErr\}\(0\)\. Here,ℓ=0\\ell=0corresponds to direct prompting, while larger values ofℓ\\ellcorrespond to estimates reconstructed from increasingly fine\-grained subgroup predictions\. Error bars indicate 90 % confidence intervals computed using 1000 bootstrap samples\. In[Figure2\(a\)](https://arxiv.org/html/2607.15277#S4.F2.sf1), we compare a selection of models at a fixed income threshold ofτ=40k\\tau=40\\mathrm\{k\}USD\. In[Figure2\(b\)](https://arxiv.org/html/2607.15277#S4.F2.sf2), we show the aggregation gain for GPT\-5\.4 across income thresholds\. Across both settings, the results show that aggregate estimates reconstructed from subgroup estimates improve over direct root\-level estimates up to moderate tree depths\.
### F\.4Disentangling the sources of aggregation error
The two panels in[Figures3\(a\)](https://arxiv.org/html/2607.15277#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2607.15277#S4.F3.sf2)are based on the depth\-six binary conditioning tree described in[SectionF\.3](https://arxiv.org/html/2607.15277#A6.SS3)\. For visual clarity, we display only the first three levels in the conditional\-estimates panel and the first four levels in the subgroup\-priors panel\.
#### Node\-wise conditional alignment\.
For[Figure3\(a\)](https://arxiv.org/html/2607.15277#S4.F3.sf1), we evaluate the quality of the LLM\-estimated conditional income distributions at each node of the tree\. The outcome is the four\-bin income distribution used in[Figure14](https://arxiv.org/html/2607.15277#A6.F14)\. For every node𝒮\\mathcal\{S\}, we compare the LLM\-estimated conditional distributionT^h\(𝒮\)\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\)to the corresponding survey\-based conditional distributionTh\(𝒮\)T\_\{h\}\(\\mathcal\{S\}\)using the normalized Wasserstein distance,𝒲¯1\(T^h\(𝒮\),Th\(𝒮\)\)\\overline\{\\mathcal\{W\}\}\_\{1\}\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\($\}\}\}\\hat\{T\}\_\{h\}\(\\mathcal\{S\}\),T\_\{h\}\(\\mathcal\{S\}\)\\vbox\{\\hbox\{\\scalebox\{1\.15\}\{$\)$\}\}\}\. The node color encodes this error\. The resulting plot illustrates that conditioning on more specific subgroups improves conditional alignment relative to the root estimate, but that the improvement is heterogeneous across the tree\. Some fine\-grained subgroups with negligible prior mass remain poorly aligned\.
#### Prior alignment\.
For[Figure3\(b\)](https://arxiv.org/html/2607.15277#S4.F3.sf2), we evaluate the quality of the subgroup priors\. For each tree levelℓ\\ell, letγℓ\\gamma\_\{\\ell\}denote the survey\-based distribution over the partition𝒫ℓ\(𝒳\)\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\), and letγ^ℓ\\hat\{\\gamma\}\_\{\\ell\}denote the corresponding LLM\-estimated prior distribution obtained with the elicitation procedure from[SectionF\.2](https://arxiv.org/html/2607.15277#A6.SS2)\. We measure prior alignment using total variation distance,TV\(γ^ℓ,γℓ\)\\operatorname\{TV\}\(\\hat\{\\gamma\}\_\{\\ell\},\\gamma\_\{\\ell\}\)\. This yields one prior\-alignment error per model and tree level\. The results show that prior estimation becomes increasingly difficult as the tree is refined\. Coarse partitions are estimated comparatively accurately, whereas deeper levels require the model to distribute probability mass across a larger number of more specific subpopulations\. Thus, aggregation involves a trade\-off\. More specific conditioning can improve the subgroup outcome estimates, but the corresponding population weights become harder to estimate accurately\.
### F\.5Variance decomposition across tree levels
The variance decomposition in[Figure3\(c\)](https://arxiv.org/html/2607.15277#S4.F3.sf3)is computed on the first four levels of the binary conditioning tree described in[SectionF\.3](https://arxiv.org/html/2607.15277#A6.SS3)\. For each node, we compute the survey\-weighted subgroup priorγ𝒮\\gamma\_\{\\mathcal\{S\}\}, the conditional meanμ𝒮≜𝔼\[Y\|X∈𝒮\]\\mu\_\{\\mathcal\{S\}\}\\triangleq\\mathbb\{E\}\[Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\], and the conditional variance𝕍\(Y\|X∈𝒮\)\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\)from the ACS data\. We then decompose the aggregate income variance into a within\-subgroup and a cross\-subgroup component\. The within\-subgroup term is the prior\-weighted average conditional variance,
Vwithin\(ℓ\)≜∑𝒮∈𝒫ℓ\(𝒳\)γ𝒮𝕍\(Y\|X∈𝒮\),V\_\{\\mathrm\{within\}\}\(\\ell\)\\triangleq\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\gamma\_\{\\mathcal\{S\}\}\\kern 1\.00006pt\\mathbb\{V\}\(Y\\kern 1\.00006pt\|\\kern 1\.00006ptX\\in\\mathcal\{S\}\),\(73\)which measures the residual heterogeneity that remains after conditioning on the attributes defining the level\-ℓ\\ellsubgroups\. The cross\-subgroup term is the prior\-weighted variance of the subgroup means,
Vcross\(ℓ\)≜∑𝒮∈𝒫ℓ\(𝒳\)γ𝒮\(μ𝒮−μ\)2withμ≜∑𝒮∈𝒫ℓ\(𝒳\)γ𝒮μ𝒮\.V\_\{\\mathrm\{cross\}\}\(\\ell\)\\triangleq\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\gamma\_\{\\mathcal\{S\}\}\\,\(\\mu\_\{\\mathcal\{S\}\}\-\\mu\)^\{2\}\\qquad\\text\{with\}\\qquad\\mu\\triangleq\\sum\_\{\\mathcal\{S\}\\in\\mathcal\{P\}\_\{\\ell\}\(\\mathcal\{X\}\)\}\\gamma\_\{\\mathcal\{S\}\}\\,\\mu\_\{\\mathcal\{S\}\}\.\(74\)By the law of total variance, these quantities satisfy𝕍\(Y\)=Vwithin\(ℓ\)\+Vcross\(ℓ\)\\mathbb\{V\}\(Y\)=V\_\{\\mathrm\{within\}\}\(\\ell\)\+V\_\{\\mathrm\{cross\}\}\(\\ell\)for every levelℓ\\ell, up to finite\-sample estimation error\. Thus, the two terms quantify how refinement shifts variation from residual within\-subgroup heterogeneity to explicit cross\-subgroup heterogeneity\.
### F\.6Self\-consistency evaluation on the binned ACS income setting
For the self\-consistency analysis in[Table2\(a\)](https://arxiv.org/html/2607.15277#A5.T2.st1), we use the binned ACS income prediction task from the running example in[Figure14](https://arxiv.org/html/2607.15277#A6.F14)\. The model is asked to estimate the income distribution over the following four bins:
- Bin 1:\[−11\.5k,1\)\[\-11\.5\\mathrm\{k\},1\)USD
- Bin 2:\[1,25k\)\[1,25\\mathrm\{k\}\)USD
- Bin 3:\[25k,60k\)\[25\\mathrm\{k\},60\\mathrm\{k\}\)USD
- Bin 4:\[60k,1849k\]\[60\\mathrm\{k\},1849\\mathrm\{k\}\]USD
These bins cover the full income range observed in the ACS sample: no individual reports an income below−11\.5k\-11\.5\\mathrm\{k\}USD or above1849k1849\\mathrm\{k\}USD\. The results in[Table2\(a\)](https://arxiv.org/html/2607.15277#A5.T2.st1)report the two benchmark scores introduced in[Section5](https://arxiv.org/html/2607.15277#S5)\. The first is the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)from[Definition2](https://arxiv.org/html/2607.15277#Thmtheorem2)\. For every admissible pair of a conditioning event and a next split attribute in𝒞SC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{SC\}\}\(\\mathcal\{A\}\), we compare the model’s direct estimate at the conditioning event to the prior\-weighted aggregate of its estimates at the two refinements induced by the split attribute\. For the attribute set𝒜\\mathcal\{A\}consisting of age and employment status, this yields six node\-level checks\. The second is the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)from[Definition4](https://arxiv.org/html/2607.15277#Thmtheorem4)\. For each of the four two\-attribute conditioning events in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{OC\}\}\(\\mathcal\{A\}\), we elicit direct estimates under both orderings of the attribute constraints and check whether the two predictions agree\. Since the target is an ordered categorical distribution, discrepancies are measured using the normalized Wasserstein distance𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}\. Each score reports the fraction of checks whose discrepancy is less than the toleranceε=0\.02\\varepsilon=0\.02\.
### F\.7Self\-consistency evaluation on the thresholded ACS income setting
For the model comparison in[Figure4](https://arxiv.org/html/2607.15277#S5.F4), we use the thresholded ACS income prediction task with thresholdτ=40k\\tau=40\\mathrm\{k\}USD\. The setting mirrors the binned income experiment in[SectionF\.6](https://arxiv.org/html/2607.15277#A6.SS6)\. We use the same conditioning tree over the attribute set𝒜\\mathcal\{A\}consisting of age and employment status\. The only change is the target variable, which is now the binary indicator of whether an individual’s income exceedsτ\\tau\.
For each model, we compute the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02\. Since the target is binary, the normalized Wasserstein distance𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}reduces to the absolute difference between the two estimated probabilities of exceeding the threshold\. As a proxy for general model capability, we report the Artificial Analysis Intelligence \(AAI\) Index\[[Artificial Analysis, 2026](https://arxiv.org/html/2607.15277#bib.bib33)\]\. Models are grouped by family and ordered within each family by their AAI Index\.
### F\.8Self\-consistency evaluation on the ACS commute time setting
For the self\-consistency analysis in[Table2\(b\)](https://arxiv.org/html/2607.15277#A5.T2.st2), we use the binned ACS commute time prediction task\. The setting mirrors the binned income experiment in[SectionF\.6](https://arxiv.org/html/2607.15277#A6.SS6)\. We use the same conditioning tree over the attribute set𝒜\\mathcal\{A\}consisting of age and employment status, and the same subgroup priors\. The only change is the target variable, which is now the commute time in minutes\. The model is asked to estimate the commute time distribution over the following four bins:
- Bin 1:\[0,1\)\[0,1\)minute
- Bin 2:\[1,15\)\[1,15\)minutes
- Bin 3:\[15,30\)\[15,30\)minutes
- Bin 4:\[30,195\]\[30,195\]minutes
These bins cover the full range of commute times observed in the ACS sample\. No individual reports a commute time longer than195195minutes\. We compute the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02, exactly as in[SectionF\.6](https://arxiv.org/html/2607.15277#A6.SS6)\. Since the target is an ordered categorical distribution, discrepancies are measured using the normalized Wasserstein distance𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}\.
### F\.9Self\-consistency evaluation on WVS opinion distributions
We evaluate self\-consistency on opinion distributions derived from the World Values Survey \(WVS\)\. The experiment reported in[Table3](https://arxiv.org/html/2607.15277#A5.T3)uses data from Canada and Indonesia\. For both countries, the evaluation is based on the attribute set𝒜\\mathcal\{A\}consisting of the following two splits:
- Level 1:Age is split into “16 – 44 years old” and “45 – 103 years old”\.
- Level 2:Income is split into “low” and “high”\.
The interpretation of “low” and “high” in the income split is left to the model\. This is sufficient for our self\-consistency checks, provided that the two options are interpreted as complementary categories that form a valid partition of the relevant population\. For each country and each of the five selected WVS questions listed in[Table8](https://arxiv.org/html/2607.15277#A10.T8), we elicit categorical response distributions at every conditioning event induced by𝒜\\mathcal\{A\}\. The corresponding prompt template is given in[SectionH\.5](https://arxiv.org/html/2607.15277#A8.SS5)\. Each question then defines a distinct prediction problem, on which we compute the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02\. For the split consistency score, we evaluate all admissible pairs of a conditioning event and a next split attribute in𝒞SC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{SC\}\}\(\\mathcal\{A\}\), comparing the direct estimate at the conditioning event to the prior\-weighted aggregate of its two refinements\. For the order consistency score, we elicit direct estimates under both orderings of the attribute constraints for each of the four two\-attribute conditioning events in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{OC\}\}\(\\mathcal\{A\}\)and check whether the two predictions agree\. Discrepancies are measured using the normalized Wasserstein distance𝒲¯1\\overline\{\\mathcal\{W\}\}\_\{1\}, and each score reports the fraction of checks whose discrepancy is less thanε=0\.02\\varepsilon=0\.02\.
### F\.10Self\-consistency evaluation of the tennis forecasting example
Consider a forecasting task in which a model is asked to estimate the probability that Roger Federer wins a match against Rafael Nadal, conditional on contextual information available before or during the match\. All estimates are elicited using the tennis forecasting prompt template in[SectionH\.6](https://arxiv.org/html/2607.15277#A8.SS6)\. At the root node, the prompt provides only the matchup itself, corresponding to a blind or unconditional forecast without additional information\. We then gradually reveal pieces of contextual information that may become available before or during the match, such as the court surface\. These conditions induce a binary conditioning tree analogous to the BCTs used in the income prediction setting, but the conditioning variables now describe a sporting event rather than a demographic subpopulation\. Concretely, the attribute set𝒜\\mathcal\{A\}consists of two binary conditions: the court surface, distinguishing clay from non\-clay courts, and the in\-match event of whether Federer wins the first set\. The former captures a pre\-match contextual factor that is highly relevant to the Federer–Nadal rivalry, while the latter represents partial information that becomes available during the match\. The same self\-consistency checks from[Section5](https://arxiv.org/html/2607.15277#S5)can then be applied, with the model’s estimated prior probabilities serving as aggregation weights\.
To generate the results in[Table1\(a\)](https://arxiv.org/html/2607.15277#S6.T1.st1), we compute the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)at toleranceε=0\.02\\varepsilon=0\.02\. Since the target outcome is binary, discrepancies are measured by the absolute difference between the corresponding win probabilities\. Each score reports the fraction of checks whose discrepancy is less thanε=0\.02\\varepsilon=0\.02\.
### F\.11Self\-consistency evaluation of the fantasy example
As a second synthetic example, we consider a reasoning task in a fictional fantasy role\-playing game\. The model is asked to estimate the probability that the main character wins an encounter against an opponent using the prompt template in[SectionH\.7](https://arxiv.org/html/2607.15277#A8.SS7)\. Unlike the survey setting, and unlike the tennis example, this task need not correspond to any real\-world data\-generating process\. The game world is imaginary, and there is no external ground\-truth distribution against which the model’s estimates can be evaluated\. This makes the example useful for highlighting a key feature of our self\-consistency checks\. They only require the model’s own direct and conditional estimates, and therefore remain meaningful even when no reference distribution exists\. At the root node, the prompt specifies only the basic combat situation\. We then refine the scenario by revealing additional contextual information about the game world\. The attribute set𝒜\\mathcal\{A\}consists of two binary conditions: the time of day, distinguishing day and night, and the wind conditions, distinguishing calm from windy combat environments\.
To generate the results in[Table1\(b\)](https://arxiv.org/html/2607.15277#S6.T1.st2), we proceed exactly as in the tennis example \([SectionF\.10](https://arxiv.org/html/2607.15277#A6.SS10)\)\. We compute the split consistency scoreSCε\(𝒜\)\\mathrm\{SC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)over the admissible checks in𝒞SC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{SC\}\}\(\\mathcal\{A\}\)and the order consistency scoreOCε\(𝒜\)\\mathrm\{OC\}\_\{\\varepsilon\}\(\\mathcal\{A\}\)over the four two\-attribute conditioning events in𝒞OC\(𝒜\)\\mathcal\{C\}\_\{\\mathrm\{OC\}\}\(\\mathcal\{A\}\)\. Consistency requires the compared estimates to agree, even though the underlying event is fictional and no ground\-truth win probability is available\. Since the target outcome is binary, discrepancies are measured by the absolute difference between the corresponding win probabilities, and each score reports the fraction of checks whose discrepancy is less thanε=0\.02\\varepsilon=0\.02\.
## Appendix GAdditional plots
This appendix collects larger plots with more detailed breakdowns\.
### G\.1Aggregation gain
We provide additional results on the aggregation gain to complement the main findings in[Section4](https://arxiv.org/html/2607.15277#S4)\. While[Figure2](https://arxiv.org/html/2607.15277#S4.F2)highlights representative settings,[Figure16](https://arxiv.org/html/2607.15277#A7.F16)reports the aggregation gain across a broader range of fixed income thresholds\. These results show that the qualitative pattern from the main text is robust\. Reconstructed aggregate estimates from subgroup conditionals are often better aligned with the ACS reference statistic than direct estimates, although the gains vary across thresholds, models, and tree levels\.
01234Level−\-100%−\-50%0%50%Aggregation gain01234Level0%25%50%75%Aggregation gainGPT\-5\.4GPT\-4o mini3\.1 Pro3 FlashOpus 4\.7Grok 4\.203\.6 PlusK2\.6
\(a\)Fixed income thresholdτ=1k\\tau=1\\mathrm\{k\}USD\.\(b\)Fixed income thresholdτ=10k\\tau=10\\mathrm\{k\}USD\.01234Level0%25%50%75%Aggregation gain01234Level−\-100%−\-50%0%50%Aggregation gain
\(c\)Fixed income thresholdτ=50k\\tau=50\\mathrm\{k\}USD\.\(d\)Fixed income thresholdτ=80k\\tau=80\\mathrm\{k\}USD\.
Figure 16:Aggregation gain across models and tree levels\.We report the aggregation gain1−AErr\(ℓ\)/AErr\(0\)1\-\\text\{AErr\}\(\\ell\)/\\text\{AErr\}\(0\)withAErr\(ℓ\)\\mathrm\{AErr\}\(\\ell\)defined in \([5](https://arxiv.org/html/2607.15277#S4.E5)\)\. Positive values indicate that reconstructing the population\-level estimate from subgroup estimates improves alignment relative to direct prompting, while negative values indicate worse alignment\. Each bar corresponds to a different model\. Error bars show 90 % confidence intervals computed by bootstrapping with 1000 samples\. The orange curve shows the average aggregation gain\.
### G\.2Reconstructed aggregate distributions across tree levels
[Figure17](https://arxiv.org/html/2607.15277#A7.F17)visualizes the population\-level ACS income distributions reconstructed from each tree level using LLM\-estimated subgroup priors\.
Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 0Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 1Income0\.00\.00\.20\.2ℙ\\mathbb\{P\}Level 2ACSLLM\[−11\.5k,1\)\[\-11\.5\\mathrm\{k\},1\)USD\[1,25k\)\[1,25\\mathrm\{k\}\)USD\[25k,60k\)\[25\\mathrm\{k\},60\\mathrm\{k\}\)USD\[60k,1849k\]\[60\\mathrm\{k\},1849\\mathrm\{k\}\]USD
Figure 17:Reconstructed aggregate income distributions across tree levels\.For each level of the binary conditioning tree in[Figure14](https://arxiv.org/html/2607.15277#A6.F14), we reconstruct the population\-level ACS income distribution by aggregating the subgroup distributions with subgroup priors\. Solid bars show the same survey\-based ground\-truth distribution of the base population in all panels, while hatched bars show the corresponding LLM\-induced reconstructed aggregate\. The left panel corresponds to the direct estimate at the root, whereas the middle and right panels show aggregates reconstructed from increasingly fine\-grained conditional estimates\. The comparison illustrates that the reconstructed aggregate becomes better aligned with the ground\-truth distribution as we move down the tree\.
## Appendix HPrompt templates
This appendix reports the exact prompt templates used to elicit probability estimates from language models\.
### H\.1Prior elicitation
YouaregivenamutuallyexclusiveandexhaustivepartitionoftheU\.S\.populationintogroupsdefinedbysociodemographicattributes\.
Foreachgroupbelow,estimatetheprobabilitythatarandomlysampledpersonfromtheU\.S\.populationbelongstothatgroup\.
\{groups\}
Important:
\-Treatthelistedgroupsasthefullpopulationpartition\.
\-Assignprobabilitymassacrossallgroups\.
\-Thereturnedprobabilitiesmustsumto1\.
\-UseyourbestdemographicknowledgeoftheU\.S\.population\.
ReturnonlyavalidJSONobjectofthefollowingform:
\{
"Group1":0\.35,
"Group2":0\.65,
…
\}
Strictoutputrules:
\-DonotincludeanytextoutsidetheJSONobject\.
\-Donotincludeexplanationsoruncertaintyintervals\.
\-Eachprobabilitymustbeasinglenumberin\[0,1\]\.
\-TheJSONobjectmustcontainexactlyoneentrypergroup\.
\-Usethegroupnamesexactlyasgiven:Group1,Group2\.
Thegroupsplaceholder is replaced by a list of groups, each specified by a structured list of sociodemographic attributes that define the corresponding subgroup in the partition\. For example:
\-sex:female
Group2
\-sex:male
### H\.2Sociodemographic prompting
Youaregivenabriefsociodemographicprofileofanindividual\.BasedontheinformationprovidedandyourgeneralknowledgeofincomedistributionsintheUnitedStates,estimatetheprobabilityofthespecifiedoutcome\.Thedataprovidedissufficienttomakeanapproximateprobabilisticestimate\.
Individualattributes:
\{attributes\}
Question:
Whatistheestimatedprobabilitythatthisperson’sannualincomeexceeds\{income\_threshold\}USD?
Outputrequirements\(strict\):
\-Returnasinglerealnumberbetween0and1\(inclusive\)\.
\-Formattheanswerexactlyas:\[\[p\]\]wherepisyourprobabilityestimate\(forexample:\[\[0\.5\]\]\)\.
In the sociodemographic formulation, theattributesplaceholder is replaced by a structured list of demographic attributes describing the target individual\. For example:
\-Sex:Female
\-Employmentstatus:Civilianemployed,atwork
\-Age:24yearsold
This formulation describes the conditioning information in the third person and avoids asking the model to adopt a role\-playing perspective\.
### H\.3Persona prompting
Pleaseanswerthefollowingquestionfromtheperspectiveofthedescribedperson\.Stayconsistentwiththeprofile,butdonotmentionthatyouarerole\-playingunlessasked\.
Persona:
\{persona\}
Question:
Whatistheestimatedprobabilitythatthisperson’sannualincomeexceeds\{income\_threshold\}USD?
Outputrequirements\(strict\):
\-Returnasinglerealnumberbetween0and1\(inclusive\)\.
\-Formattheanswerexactlyas:\[\[p\]\]wherepisyourprobabilityestimate\(forexample:\[\[0\.5\]\]\)\.
In the persona formulation, thepersonaplaceholder is replaced by a free\-text description that conveys the same demographic information in a first\-person role\-playing style\. For example, the profile may read: “You are a person who lives in the United States\. Your sex is Female\.”
### H\.4Partition\-then\-aggregate prompting
BasedonyourgeneralknowledgeofincomedistributionsintheUnitedStates,estimatetheprobabilityofthespecifiedoutcome\.Pleasemakeanapproximateprobabilisticestimate\.
Question:
Whatistheestimatedprobabilitythatarandomlyselectedperson’sannualincomeexceeds\{income\_threshold\}USD?
Beforegivingyourfinalanswer,reasonfrommicrotomacro:
1\.IdentifyseveralrelevantsubpopulationswithintheU\.S\.populationthatmaydifferinincome\.
2\.Foreachsubpopulation,considerbothitsapproximateprevalenceintheU\.S\.populationanditslikelyprobabilityofexceedingtheincomethreshold\.
3\.Combinethesesubgroup\-levelconsiderationsintoasingleaggregateestimatefortheU\.S\.population\.
Outputrequirements\(strict\):
\-Returnasinglerealnumberbetween0and1\(inclusive\)\.
\-Formattheanswerexactlyas:\[\[p\]\]wherepisyourprobabilityestimate\(forexample:\[\[0\.5\]\]\)\.
This prompt asks the model to estimate a population\-level income probability by explicitly reasoning through relevant U\.S\. subpopulations, their prevalence, and their subgroup\-specific probabilities\. Unlike our tree\-based aggregation prompts, the relevant subpopulations are not specified in advance but are instead chosen by the model\.
### H\.5Opinion distribution elicitation
Youareestimatingthedistributionofsurveyresponsesforademographicsubpopulation\.Thesubpopulationischaracterizedbythefollowingattributes:
\{attribute\_list\}
Surveyquestion:
\{question\}
Answeroptions:
\{answer\_option\_list\}
Task:
Estimatetheprobabilitydistributionovertheansweroptionsforarandomlyselectedindividualfromthissubpopulation\.
Strictoutputrules:
\-ReturnonlyvalidJSON\.
\-Useexactlythefollowingkeys:\{answer\_option\_keys\}\.
\-Eachvaluemustbeasinglenumberin\[0,1\]\.
\-Thevaluesmustsumto1\.
\-Usedecimalprobabilities,notpercentages\.
\-Donotincludeexplanations,comments,uncertaintyintervals,oranytextoutsidetheJSONobject\.
ReturnonlyavalidJSONobjectofthefollowingform\.Thenumbersbelowareillustrativeonlyandmustbereplacedbyyourestimatedprobabilities:
\{example\_output\}
For the WVS opinion\-distribution experiments, theattribute\_listplaceholder is replaced by the demographic attributes defining the target subpopulation,questionis replaced by the survey question text, andanswer\_option\_listlists the available response options\. The model is instructed to return a complete categorical probability distribution over the answer options as a JSON object\. The placeholdersanswer\_option\_keysandexample\_outputensure that the returned object has a fixed schema across models and prompts\.
### H\.6Tennis forecasting task
Youareestimatingtheoutcomeofatennismatch\.
Match:
RogerFedererplaysagainstRafaelNadal\.
Thematchcontextischaracterizedbythefollowingattributes:
\{attribute\_list\}
Forecastingquestion:
WhatistheprobabilitythatRogerFedererwinsthematch?
Task:
EstimatetheprobabilitythatRogerFedererwinsthematchunderthespecifiedcontext\.
Strictoutputrules:
\-Returnasinglerealnumberbetween0and1,inclusive\.
\-Usedecimalprobabilities,notpercentages\.
\-Formattheanswerexactlyas:\[\[p\]\],wherepisyourprobabilityestimate\.
\-Donotincludeexplanations,comments,uncertaintyintervals,oranytextoutsidetherequiredformat\.
Returnonlyananswerofthefollowingform\.Thenumberbelowisillustrativeonlyandmustbereplacedbyyourestimatedprobability:\[\[0\.5\]\]
### H\.7Fictional fantasy combat task
Youareestimatingtheoutcomeofafantasycombatencounter\.
Amaincharacterisengagedincombatagainstanenemyinanimaginaryfantasysetting\.Theworldhasaday\-nightcycleandvaryingweatherconditions\.
Thecurrentencountercontextischaracterizedbythefollowingattributes:
\{attribute\_list\}
Forecastingquestion:
Whatistheprobabilitythatthemaincharacterwinstheencounter?
Task:
Estimatetheprobabilitythatthemaincharacterwinstheencounterunderthespecifiedcontext\.
Strictoutputrules:
\-Returnasinglerealnumberbetween0and1,inclusive\.
\-Usedecimalprobabilities,notpercentages\.
\-Formattheanswerexactlyas:\[\[p\]\],wherepisyourprobabilityestimate\.
\-Donotincludeexplanations,comments,uncertaintyintervals,oranytextoutsidetherequiredformat\.
Returnonlyananswerofthefollowingform\.Thenumberbelowisillustrativeonlyandmustbereplacedbyyourestimatedprobability:\[\[0\.5\]\]
### H\.8Conditional estimates from decision models
To elicit conditional estimates from the evaluated decision models, we use prompts comprising a question, a context, and a set of answer options, as illustrated below\. The context attributes and their values are illustrative and may vary across the prompts presented to the models\.
Isthisperson’sannualincomegreaterthan40000USD?Estimatetheprobabilityunderthespecifiedcontext\.
ThepersonisanindividualintheUnitedStateswiththefollowingattributes:
\-age:40–60yearsold
\-classofworker:Currentlyemployedorself\-employed
\-usualnumberofhoursworkedperweek:40–46hoursperweek
\{
"A":"Yes,theirannualincomeisgreaterthan40000USD\.",
"B":"No,theirannualincomeisatmost40000USD\."
\}
### H\.9Prior estimates from decision models
Prior estimates are elicited jointly for all nodes in a given tree level, with each node represented by an answer option describing its defining attributes\.
Whichgroupdoesarandomlysampledpersonbelongto?Estimatetheprobabilityofeachoptionunderthespecifiedcontext\.
ApersonissampleduniformlyatrandomfromtheU\.S\.population\.UseyourbackgroundknowledgeaboutthedemographiccompositionoftheUnitedStates\.
\{
"Group1":"Thepersonhasthefollowingattributes:…",
"Group2":"Thepersonhasthefollowingattributes:…"
\}
## Appendix IACS attributes
[Table7](https://arxiv.org/html/2607.15277#A9.T7)summarizes the ACS attributes used to define the conditioning variables in our experiments\.
Table 7:ACS attributes used for demographic conditioning\.We list the ACS variables used to construct the conditioning trees in our experiments, together with their descriptions and possible values\. Integer\-valued variables are reported by their observed range, while finite categorical variables are reported using their ACS response codes\. For the demographic splits, we augment each value list with ann/aclass to represent missing values in the ACS survey data\.
## Appendix JWVS questions
For completeness,[Table8](https://arxiv.org/html/2607.15277#A10.T8)reports the World Values Survey questions used in our experiments, including their original WVS identifiers, question wording, and ordinal answer scales\.
Table 8:Selected World Values Survey questions\.We report the five WVS questions used in our experiments, together with their original WVS identifiers, question text, and ordinal answer scales\.相似文章
分区、提示、聚合:语言模型中的统计自一致性
本文提出了一个框架来测试LLM估计是否在不同子群体间遵循统计自一致性(全概率定理),发现了广泛的违反以及“宏观谬误”,即细粒度估计与人类数据的一致性更好。
VALCON:通过双角色变换评估并增强大语言模型的价值观一致性
该论文提出VALCON框架,通过生成者/评判者双重角色变体来基准测试并提升大语言模型的价值一致性,实验揭示28个LLM普遍存在跨角色不一致问题,并提出基于表征的干预方法加以缓解。
泛化即稳定性,而非准确率:大语言模型的多轴向评估
该论文提出 Stability-Aware Generalization Objective(SAGO)框架,主张 LLM 的泛化应评估为稳定性而非准确率,通过生成一致性、内部激活、置信度和响应镜像等多维度变异测量,揭示了现有模型普遍存在且与排名相关的泛化不稳定问题。
大语言模型在患者证据演变过程中更新临床判断的可靠性不足
研究评估了大语言模型在患者证据随时间演变时更新临床判断的可靠性,发现模型存在对恶化证据反应过度、受先验判断因果影响等失败模式,并提出EVLU方法来识别更可靠但覆盖更少的判断修订,确立了纵向信念更新作为LLM可靠性的独立评估维度。
LLM群体智慧:语言模型集成中的聚合与污染
本文研究了将多个LLM的概率估计进行聚合是否会产生群体智慧效应,发现学习型聚合器优于单个模型,并且训练截止期污染是此类评估中普遍存在的混淆因素。