Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning

# Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning
Source: [https://arxiv.org/html/2608.20564](https://arxiv.org/html/2608.20564)
Abhijith Babu1, Ramneet Kaur2, Vishal Pramanik3, Olivera Kotevska4, Nathaniel D\. Bastian5

Susmit Jha2, Sunny Raj6, Yanzhao Wu1, Sumit Kumar Jha3,\*, Anirban Roy2

1Knight Foundation School of Computing and Information Sciences, Florida International University 2SRI International3Department of Computer & Information Science & Engineering, University of Florida 4Oak Ridge National Laboratory5Army Cyber Institute, United States Military Academy 6Department of Computer Science and Engineering, Oakland University \*Corresponding author: sumit\.jha@ufl\.edu

###### Abstract

Multi\-agent LLM systems can improve reasoning by pooling diverse perspectives, but their effectiveness depends on coordinating communication, particularly in hidden\-profile settings where each agent holds only part of the evidence required for a correct decision\. Existing protocols, including fixed schedules, round\-robin exchange, and unstructured debate, provide no guarantee that a conversational action is appropriate\. We proposeConsilience, an inference\-time orchestration framework that both steers and certifies multi\-agent communication under distributed private information\. At each turn, Consilience summarizes the discussion using a compact state capturing uncertainty, disagreement, evidence gain, redundancy, and premature consensus, then selects both a communication intervention \(challenge, clarify, seek evidence, or route\) and an appropriate speaker\. Its central contribution is a round\-wise conformal calibration procedure that provides a distribution\-free, finite\-sample guarantee: at each discussion round, conditional on reaching that round, the one\-step regret of a controller’s proposed action is bounded by a calibrated threshold with marginal probability at least1−α1\-\\alpha; an acceptance mechanism enforces the same guarantee for the executed action by replacing inadmissible proposals\. On HiddenBench\-style hidden\-profile tasks spanning 12 open and closed weight language models, Consilience improves decision accuracy and communication efficiency over fixed and unstructured discussion protocols, sometimes surpassing a full\-information baseline where every agent observes all evidence\. These results demonstrate that certified adaptive communication control can be more valuable than increasing information availability, providing a practical mechanism for reliable multi\-agent LLM coordination\.

## 1Introduction

Large language models \(LLMs\) are increasingly deployed as multi\-agent systems in which several agents propose solutions, exchange arguments, critique one another, and jointly produce a decision[6](https://arxiv.org/html/2608.20564#bib.bib10);[12](https://arxiv.org/html/2608.20564#bib.bib11);[23](https://arxiv.org/html/2608.20564#bib.bib12);[22](https://arxiv.org/html/2608.20564#bib.bib16);[10](https://arxiv.org/html/2608.20564#bib.bib15)\. Such systems have shown promise in mathematical reasoning, question answering, software development, planning, and tool\-assisted problem solving\. Their appeal stems from the idea that multiple agents can contribute complementary reasoning, identify errors overlooked by individual models, and produce more reliable collective decisions\. However, recent studies question whether discussion alone consistently delivers these benefits\. Multi\-agent deliberation can duplicate the capabilities of a well\-prompted single agent, amplify correlated errors, or converge prematurely on persuasive but incorrect conclusions[21](https://arxiv.org/html/2608.20564#bib.bib13);[12](https://arxiv.org/html/2608.20564#bib.bib11)\. Simply increasing the number of agents or discussion rounds therefore does not guarantee better collective reasoning\.

This limitation becomes especially consequential when task\-relevant information is distributed across agents\. Many collaborative decisions in medicine, intelligence analysis, and organizational planning involve participants who observe different pieces of evidence\. No individual has access to the complete information required for a correct decision; the group succeeds only if its members communicate and integrate their complementary knowledge\. This setting is captured by thehidden\-profile paradigm, originally introduced in social psychology to study information pooling in group decision making[17](https://arxiv.org/html/2608.20564#bib.bib3);[18](https://arxiv.org/html/2608.20564#bib.bib4)\. In a hidden profile, information shared by all group members supports a plausible but suboptimal alternative, whereas uniquely held information, when pooled across members, reveals the correct choice[17](https://arxiv.org/html/2608.20564#bib.bib3);[14](https://arxiv.org/html/2608.20564#bib.bib7);[15](https://arxiv.org/html/2608.20564#bib.bib8)\.

Although discussion should theoretically recover distributed evidence, human groups consistently exhibit a common\-information bias: information shared by several members is more likely to be mentioned, repeated, and reinforced than information known to only one member[17](https://arxiv.org/html/2608.20564#bib.bib3);[16](https://arxiv.org/html/2608.20564#bib.bib5);[7](https://arxiv.org/html/2608.20564#bib.bib6)\. As a result, discussion often reinforces initial preferences instead of correcting them\. A meta\-analysis of hidden\-profile studies found that groups discussed substantially more common than unique information and were significantly less likely to identify the correct solution under hidden\- than full\-information conditions[14](https://arxiv.org/html/2608.20564#bib.bib7)\. Effective hidden\-profile reasoning therefore requires more than allowing participants to speak: groups must surface unshared evidence, consider dissenting perspectives, revise premature beliefs, and determine when consensus is sufficiently supported to justify termination[14](https://arxiv.org/html/2608.20564#bib.bib7);[15](https://arxiv.org/html/2608.20564#bib.bib8)\.

Recent work shows that LLM\-based groups reproduce many of these collective reasoning failures\. HiddenBench\([11](https://arxiv.org/html/2608.20564#bib.bib9)\)formalizes the hidden\-profile paradigm for multi\-agent LLM evaluation using 65 tasks derived from custom scenarios, prior human studies, and automatically generated decision problems\. Each agent receives shared task information together with an asymmetric private clue, and the agents must deliberate before making a collective decision\. Across 15 models from four language\-model families, HiddenBench finds persistent failures to integrate distributed evidence: agents can collectively possess sufficient information yet still converge on an incorrect answer, and model scale or general reasoning strength does not reliably predict successful information pooling[11](https://arxiv.org/html/2608.20564#bib.bib9)\. HiddenBench, therefore, exposes a gap not directly addressed by the conventional reasoning benchmarks\. The bottleneck is not only whether an LLM can reason from the evidence it observes, but whether a group can control its communication so that the right evidence is elicited, routed, and incorporated before consensus forms\.

Existing multi\-agent LLM methods provide only partial solutions\. Debate frameworks improve reasoning through critique and competing arguments[6](https://arxiv.org/html/2608.20564#bib.bib10);[12](https://arxiv.org/html/2608.20564#bib.bib11);[23](https://arxiv.org/html/2608.20564#bib.bib12); role\-based frameworks coordinate specialized agents[10](https://arxiv.org/html/2608.20564#bib.bib15);[22](https://arxiv.org/html/2608.20564#bib.bib16);[8](https://arxiv.org/html/2608.20564#bib.bib17); and coordination methods rely on predefined interaction topologies, voting, judges, or aggregation mechanisms[3](https://arxiv.org/html/2608.20564#bib.bib14);[4](https://arxiv.org/html/2608.20564#bib.bib18);[25](https://arxiv.org/html/2608.20564#bib.bib19)\. These approaches work well when agents share most of the available information\. Hidden\-profile tasks, however, require adaptive communication control: the system must decide whether to elicit missing evidence, clarify disagreement, challenge unsupported conclusions, route the discussion, or terminate deliberation\. Fixed round\-robin protocols\([11](https://arxiv.org/html/2608.20564#bib.bib9)\)cannot adapt to the evolving discussion state, while unconstrained LLM orchestrators may make unreliable control decisions\.

Moreover, existing debate and orchestration methods generally provide no statistical guarantee that the communication action selected at a given state is appropriate\. Such guarantees matter because communication decisions shape the information available to the group: selecting the wrong speaker, repeating shared evidence, or voting too early can reinforce an incorrect consensus, while unnecessary discussion increases cost and risks distraction or context degradation\. Communication control should therefore be treated as a sequential decision problem rather than a fixed prompting protocol\. At each round, the controller must use the agents’ evolving beliefs and public transcript to select an intervention that improves the collective decision state while avoiding unnecessary communication\. Our contributions are:

- •Closed\-loop communication control\.We pose hidden\-profile reasoning as a sequential communication\-control problem and introduce Consilience, a framework that uses the evolving discussion state to select, conformally certify, and route communication interventions, enabling closed\-loop coordination among LLM agents\.
- •Conformal reliability\.We propose a round\-wise conformal calibration framework that certifies communication actions using one\-step regret, providing finite\-sample, distribution\-free regret guarantees and replacing inadmissible proposals with conformally certified alternatives\.
- •Comprehensive evaluation\.We evaluate Consilience across 12 open\- and closed\-weight language models on HiddenBench[11](https://arxiv.org/html/2608.20564#bib.bib9)and an LLM\-generated benchmark based on the GroupTravelBench task design[5](https://arxiv.org/html/2608.20564#bib.bib20)\. We compare against hidden\-information, round\-robin, and full\-information baselines, while analyzing communication efficiency, conformal calibration, speaker routing, controller generalization, heterogeneous agents, and key ablations\.

![Refer to caption](https://arxiv.org/html/2608.20564v1/Figures/img5.png)Figure 1:Proposed frameworkConsilience: \(1\) Each agent receives shared task information and a private clue\. \(2\) Agents report beliefs conditioned on the public transcript, which Consilience aggregates with transcript\-level signals into the collective statests\_\{t\}\. \(3\) A controller proposes a communication action, and round\-wise conformal acceptance retains or replaces it using calibrated one\-step regret\. \(4\) An action\-conditional router selects the speaker, whose message updates the transcript and agent beliefs\. The loop repeats until termination, followed by plurality voting with confidence\-based tie\-breaking\.
## 2Problem Formulation

We consider hidden\-profile multi\-agent decision tasks in which the information needed to identify the correct answer is distributed across multiple agents\. A task is a tuple\(x,𝒴,Ishared,\{Ii\}i=1N\)\(x,\\mathcal\{Y\},I\_\{\\mathrm\{shared\}\},\\\{I\_\{i\}\\\}\_\{i=1\}^\{N\}\), wherexxis the task description,𝒴=\{y1,…,yK\}\\mathcal\{Y\}=\\\{y\_\{1\},\\ldots,y\_\{K\}\\\}is a finite answer set,IsharedI\_\{\\mathrm\{shared\}\}is visible to every agent, andIiI\_\{i\}is visible only to agentii\. Thus, agentiiinitially observes\(x,𝒴,Ishared,Ii\)\(x,\\mathcal\{Y\},I\_\{\\mathrm\{shared\}\},I\_\{i\}\), but notIjI\_\{j\}forj≠ij\\neq i\.

The agents communicate through a shared public transcriptℋt=\(m1,…,mt\)\\mathcal\{H\}\_\{t\}=\(m\_\{1\},\\ldots,m\_\{t\}\), withℋ0=∅\\mathcal\{H\}\_\{0\}=\\varnothing\. Private evidence becomes available to the group only when an agent communicates it on the public transcript\. The goal is to produce a collective answery^∈𝒴\\widehat\{y\}\\in\\mathcal\{Y\}while eliciting and integrating the relevant private evidence with limited communication\. Consilience treats this process as closed\-loop communication control: the controller chooses what kind of action intervention is needed, and a separate action\-conditional router chooses which agent should carry it out\. Neither the controller nor the router answers the task itself or generates evidence on an agent’s behalf\. The final answer is produced from the quorum among the LLM agents from their beliefs formed from the information on the public transcript\.

## 3Consilience: Conformally Calibrated Communication for Multi\-Agent Systems

### 3\.1Method Overview

Figure[1](https://arxiv.org/html/2608.20564#S1.F1)summarizes the Consilience loop\.

##### Design principle\.

LetY∈𝒴Y\\in\\mathcal\{Y\}denote the latent correct answer\. An ideal controller would choose the intervention that maximizes the expected quality of the next collective belief while penalizing communication:

at⋆=argmaxa∈𝒜𝔼\[U\(bt\+1\)−λC\(ℋt\+1\)∣st,a\],a\_\{t\}^\{\\star\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\mathbb\{E\}\\\!\\left\[U\(b\_\{t\+1\}\)\-\\lambda C\(\\mathcal\{H\}\_\{t\+1\}\)\\mid s\_\{t\},a\\right\],\(1\)wherebt\+1b\_\{t\+1\}is the collective posterior induced after the next message,U⁡\(b\)=−H⁡\(b\)U\(b\)=\-H\(b\)is negative posterior entropy, andC⁡\(ℋ\)C\(\\mathcal\{H\}\)is communication cost\. Exact optimization of Equation \([1](https://arxiv.org/html/2608.20564#S3.E1)\) is intractable because each action can induce many possible language\-model responses and future discussion trajectories\. Consilience, therefore, uses an observable collective state and a tractable one\-step objective to propose an intervention, round\-wise conformal acceptance to screen that proposal, and an action\-conditional router to select its speaker\. The resulting message updates the public transcript and agent beliefs, closing the loop\. The following subsections formalize state construction, action proposal, conformal acceptance, speaker routing, and termination\.

#### 3\.1\.1Stage 1: Belief Elicitation and State Construction

After observingℋt\\mathcal\{H\}\_\{t\}, each agent reports

pi\(t\)=\[pi\(t\)​\(y1\),…,pi\(t\)​\(yK\)\],∑k=1Kpi\(t\)​\(yk\)=1\.p\_\{i\}^\{\(t\)\}=\\left\[p\_\{i\}^\{\(t\)\}\(y\_\{1\}\),\\ldots,p\_\{i\}^\{\(t\)\}\(y\_\{K\}\)\\right\],\\qquad\\sum\_\{k=1\}^\{K\}p\_\{i\}^\{\(t\)\}\(y\_\{k\}\)=1\.This is the agent’s reported belief conditional on\(x,Ishared,Ii,ℋt\)\(x,I\_\{\\mathrm\{shared\}\},I\_\{i\},\\mathcal\{H\}\_\{t\}\); it is not assumed to be a calibrated Bayesian posterior\. The aggregate belief is the arithmetic mean

p¯\(t\)=1N​∑i=1Npi\(t\)\.\\bar\{p\}^\{\(t\)\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{\(t\)\}\.\(2\)Consilience represents the current discussion by

st=\[Ht,Dt,Rt,Gt,Ct,Pt\],s\_\{t\}=\\left\[H\_\{t\},D\_\{t\},R\_\{t\},G\_\{t\},C\_\{t\},P\_\{t\}\\right\],\(3\)whose components are defined as:

##### Group uncertainty \(HtH\_\{t\}\)

We use the normalized entropy of the aggregate belief:

Ht=−∑k=1Kp¯\(t\)​\(yk\)​log⁡p¯\(t\)​\(yk\)log⁡K\.H\_\{t\}=\-\\frac\{\\sum\_\{k=1\}^\{K\}\\bar\{p\}^\{\(t\)\}\(y\_\{k\}\)\\log\\bar\{p\}^\{\(t\)\}\(y\_\{k\}\)\}\{\\log K\}\.Thus, larger values ofHtH\_\{t\}indicate a less decisive group belief\.

##### Inter\-agent disagreement \(DtD\_\{t\}\)

Aggregate confidence can conceal incompatible individual beliefs\. We measure disagreement by

Dt=1N∑i=1NJSD\(pi\(t\)∥p¯\(t\)\),D\_\{t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\operatorname\{JSD\}\\\!\\left\(p\_\{i\}^\{\(t\)\}\\parallel\\bar\{p\}^\{\(t\)\}\\right\),whereJSD\\operatorname\{JSD\}is the Jensen–Shannon divergence[13](https://arxiv.org/html/2608.20564#bib.bib2)\. A largeDtD\_\{t\}indicates that some agents interpret the available evidence differently from the group\.

##### Message redundancy \(RtR\_\{t\}\) and evidence gain \(GtG\_\{t\}\)

Let

Rt=ϕR​\(mt,ℋt−1\)andGt=ϕG​\(mt,ℋt−1,𝒴\),R\_\{t\}=\\phi\_\{R\}\(m\_\{t\},\\mathcal\{H\}\_\{t\-1\}\)\\quad\\text\{and\}\\quad G\_\{t\}=\\phi\_\{G\}\(m\_\{t\},\\mathcal\{H\}\_\{t\-1\},\\mathcal\{Y\}\),where the fixed estimatorsϕR,ϕG∈\[0,1\]\\phi\_\{R\},\\phi\_\{G\}\\in\[0,1\]are applied identically across controllers\. The redundancy estimatorϕR\\phi\_\{R\}combines lexical overlap with an LLM assessment of whethermtm\_\{t\}repeats previously public information\. The evidence\-gain estimatorϕG\\phi\_\{G\}assesses whethermtm\_\{t\}contributes a previously unshared, decision\-relevant fact that supports or eliminates candidate answers\. HigherRtR\_\{t\}indicates more repetition, whereas higherGtG\_\{t\}indicates a more informative contribution\. We setR0=G0=0R\_\{0\}=G\_\{0\}=0\.

##### Communication cost\(Ct\)\(C\_\{t\}\)

We use normalized transcript length:

Ct=Words⁡\(ℋt\)1000\.C\_\{t\}=\\frac\{\\operatorname\{Words\}\(\\mathcal\{H\}\_\{t\}\)\}\{1000\}\.

##### Premature consensus \(OPENPt\)P\_\{t\}\)

Letpt⋆=maxy∈𝒴⁡p¯\(t\)​\(y\)p\_\{t\}^\{\\star\}=\\max\_\{y\\in\\mathcal\{Y\}\}\\bar\{p\}^\{\(t\)\}\(y\)\. We distinguish supported agreement from early anchoring as

Pt=pt⋆​\(1−Gt\)​\[pt⋆\>0\.70,Gt<0\.30,t<K\+2\]\.P\_\{t\}=p\_\{t\}^\{\\star\}\(1\-G\_\{t\}\)\\,\\mathbf\{1\}\\\!\\left\[p\_\{t\}^\{\\star\}\>0\.70,\\;G\_\{t\}<0\.30,\\;t<K\+2\\right\]\.The score is nonzero when the group becomes highly confident early in the discussion despite weak recent evidence\.

##### Surrogate discussion loss\.

The state variables define

J⁡\(st\)=αH​Ht\+αD​Dt\+αR​Rt−αG​Gt\+αC​Ct\+αP​Pt\.J\(s\_\{t\}\)=\\alpha\_\{H\}H\_\{t\}\+\\alpha\_\{D\}D\_\{t\}\+\\alpha\_\{R\}R\_\{t\}\-\\alpha\_\{G\}G\_\{t\}\+\\alpha\_\{C\}C\_\{t\}\+\\alpha\_\{P\}P\_\{t\}\.Lower values are preferred\. The coefficients are fixed design hyperparameters, selected before evaluation and held constant across controller variants and test tasks\. The objective is a local control signal, not a substitute for answer correctness\. To summarize, Stage 1: the current transcript and private agent contexts produce the observable statests\_\{t\}and its scalar discussion lossJ⁡\(st\)J\(s\_\{t\}\)\. Stage 2 uses these to propose the next intervention\.

#### 3\.1\.2Stage 2: Communication\-Action Proposal

##### Communication\-action space

Consilience uses four communication interventions:

𝒜comm=\{Challenge,Clarify,SeekEvidence,Route\}\.\\mathcal\{A\}\_\{\\mathrm\{comm\}\}=\\\{\\textsc\{Challenge\},\\textsc\{Clarify\},\\textsc\{SeekEvidence\},\\textsc\{Route\}\\\}\.Controllers that can terminate adaptively additionally includeVote\. All controller variants use the same action definitions and message templates\.

- •Challengeasks the speaker to examine the current leading answer, identify unsupported assumptions, and provide potentially contradictory evidence\.
- •Clarifyasks an agent with a divergent belief to explain the private evidence/interpretation for that disagreement\.
- •SeekEvidencerequests one previously unshared, decision\-relevant fact and an explanation of how it supports or eliminates candidate answers\.
- •Routeis the evidence\-sharing intervention\. It asks an under\-participating agent for concise discriminative facts\.
- •Vote, when available, terminates deliberation and invokes the final voting rule\.

Controller variants\.Every controller mapsst\{s\_\{t\}\}from Stage 1 to a proposal actionat,0=πm​\(st\)a\_\{t,0\}=\\pi\_\{m\}\(s\_\{t\}\)\. The routing and message\-generation procedures remain fixed, so controller comparisons isolate action proposal\.

##### Rule\-based controller

The interpretable controller applies fixed thresholds to the state variables\. It selectsChallengewhen confidence is high but recent evidence is weak,Clarifywhen disagreement is high,SeekEvidencewhen communication is redundant or uninformative, andRouteotherwise\. It permits voting only after a minimum number of rounds, when the leading belief is sufficiently strong, disagreement is low, and premature consensus is absent\.

##### Learned controller

Exhaustive training rollouts evaluate every admissible action from each encountered state using the Stage\-3 counterfactual reward and record the vector

\[rt​\(st,a1\),…,rt​\(st,a\|𝒜comm\|\)\]\.\\left\[r\_\{t\}\(s\_\{t\},a\_\{1\}\),\\ldots,r\_\{t\}\(s\_\{t\},a\_\{\|\\mathcal\{A\}\_\{\\mathrm\{comm\}\}\|\}\)\\right\]\.A multilayer perceptronfθ:ℝ6→ℝ\|𝒜comm\|f\_\{\\theta\}:\\mathbb\{R\}^\{6\}\\rightarrow\\mathbb\{R\}^\{\|\\mathcal\{A\}\_\{\\mathrm\{comm\}\}\|\}with one 12\-unit ReLU hidden layer is trained to predict this vector\. At inference time, it proposes

πMLP​\(st\)=arg⁡maxa∈𝒜comm​fθ​\(st\)a\.\\pi\_\{\\mathrm\{MLP\}\}\(s\_\{t\}\)=\\arg\\max\_\{a\\in\\mathcal\{A\}\_\{\\mathrm\{comm\}\}\}f\_\{\\theta\}\(s\_\{t\}\)\_\{a\}\.

##### LLM\-based controller

The LLM controller receives the task, candidate answers, public transcript, state vector, agent beliefs, and speaker counts, and selects one action from the admissible action space\. A second variant includesVote, enabling adaptive termination\. Speaker selection remains separate\. Regardless of controller type, Stage 2 outputs only the proposed actionat,0a\_\{t,0\}\. Stage 3 evaluates that proposal against the other admissible actions and determines the actionata\_\{t\}that may actually be executed\.

#### 3\.1\.3Stage 3: Counterfactual Evaluation and Conformal Acceptance

##### Conformal prediction

[20](https://arxiv.org/html/2608.20564#bib.bib1)provide distribution\-free, finite\-sample marginal coverage for exchangeable non\-conformity scores on calibration datapoints\(Q1,…,Qn\)\(Q\_\{1\},\\ldots,Q\_\{n\}\)and the test inputQt​e​s​tQ\_\{test\}\. Specifically, ifQ1,…,Qn,QtestQ\_\{1\},\\ldots,Q\_\{n\},Q\_\{\\mathrm\{test\}\}are exchangeable and are produced by the same fixed scoring procedure, then, fork=⌈\(n\+1\)​\(1−α\)⌉k=\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil, andϵ\\epsilonas thekk\-th smallest calibration score, we have the following probabilistic guarantees:ℙ⁡\(Qtest≤ϵ\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Q\_\{\\mathrm\{test\}\}\\leq\\epsilon\\right\)\\geq 1\-\\alpha\.

##### Counterfactual action outcomes

For a nonterminal actionaaapplied at statests\_\{t\}, the fixed Stage\-4 router selects a speaker, the speaker generates a message, all agents update their beliefs, and Consilience obtains a candidate next statest\+1as\_\{t\+1\}^\{a\}\. All candidate actions are evaluated independently from the same frozen pre\-action state; only the finally accepted branch is committed to the public transcript\. The one\-step improvement isrt​\(st,a\)=J⁡\(st\)−J⁡\(st\+1a\)\.r\_\{t\}\(s\_\{t\},a\)=J\(s\_\{t\}\)\-J\(s\_\{t\+1\}^\{a\}\)\.We propose one\-step regret as the non\-conformity score:

ℛt​\(st,a\)=maxa′∈𝒜tm​\(st\)⁡rt​\(st,a′\)−rt​\(st,a\)\\displaystyle\\mathcal\{R\}\_\{t\}\(s\_\{t\},a\)=\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\_\{t\}^\{m\}\(s\_\{t\}\)\}r\_\{t\}\(s\_\{t\},a^\{\\prime\}\)\-r\_\{t\}\(s\_\{t\},a\)=J⁡\(st\+1a\)−mina′∈𝒜tm​\(st\)⁡J⁡\(st\+1a′\),\\displaystyle=J\(s\_\{t\+1\}^\{a\}\)\-\\min\_\{a^\{\\prime\}\\in\\mathcal\{A\}\_\{t\}^\{m\}\(s\_\{t\}\)\}J\(s\_\{t\+1\}^\{a^\{\\prime\}\}\),where𝒜tm​\(st\)\\mathcal\{A\}\_\{t\}^\{m\}\(s\_\{t\}\)is the set of actions admissible for controllermmat roundtt\. Henceℛt​\(st,a\)≥0\\mathcal\{R\}\_\{t\}\(s\_\{t\},a\)\\geq 0, and at least one admissible action has regret zero\. Terminal actions, when available, are evaluated using the same fixed terminal\-state scoring rule in calibration and testing\.

##### Offline round\-wise calibration

For a fixed controller methodmm, define the non\-conformity score of its proposal at roundttas

Qtm​\(s\)=ℛt​\(s,πm​\(s\)\)\.Q\_\{t\}^\{m\}\(s\)=\\mathcal\{R\}\_\{t\}\\\!\\left\(s,\\pi\_\{m\}\(s\)\\right\)\.\(4\)Calibration uses complete trajectories generated by the fixed controller, routing policy, model, prompts, and score\-generating procedure\. Because some trajectories terminate early, only calibration trajectories that reach roundttcontribute to the round\-ttmultiset

𝒬calm,\(t\)=\{Qtm​\(stτ\):τ∈𝒟cal​reaches round​t\}\.\\mathcal\{Q\}\_\{\\mathrm\{cal\}\}^\{m,\(t\)\}=\\left\\\{Q\_\{t\}^\{m\}\(s\_\{t\}^\{\\tau\}\):\\tau\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}\\text\{ reaches round \}t\\right\\\}\.Letnt=\|𝒬calm,\(t\)\|n\_\{t\}=\|\\mathcal\{Q\}\_\{\\mathrm\{cal\}\}^\{m,\(t\)\}\|andkt=⌈\(nt\+1\)​\(1−α\)⌉\.k\_\{t\}=\\left\\lceil\(n\_\{t\}\+1\)\(1\-\\alpha\)\\right\\rceil\.The thresholdϵtm\\epsilon\_\{t\}^\{m\}is thektk\_\{t\}\-th smallest calibration score\.

##### Online conformal acceptance

At test time, non\-conformity score \(or regret\) for the controller’s proposed action is computed, and if it is within the conformal thresholdϵtm\\epsilon\_\{t\}^\{m\}, then it is passed to the router for stage 4\. Otherwise, the near\-optimal conformal action setΓtm​\(s\)\\Gamma\_\{t\}^\{m\}\(s\)is generated by executing all possible actions:

Γtm​\(s\)=\{a∈𝒜tm​\(s\):ℛt​\(s,a\)≤ϵtm\}\.\\Gamma\_\{t\}^\{m\}\(s\)=\\left\\\{a\\in\\mathcal\{A\}\_\{t\}^\{m\}\(s\):\\mathcal\{R\}\_\{t\}\(s,a\)\\leq\\epsilon\_\{t\}^\{m\}\\right\\\}\.\(5\)Here, Consilience applies a fixed fallbackSelect\\operatorname\{Select\}, which returns a minimum\-regret member ofΓtm​\(s\)\\Gamma\_\{t\}^\{m\}\(s\), with ties resolved by a fixed action ordering:

Select⁡\(Γtm​\(s\)\)∈arg⁡mina∈Γtm​\(s\)​ℛt​\(s,a\)\.\\operatorname\{Select\}\(\\Gamma\_\{t\}^\{m\}\(s\)\)\\in\\arg\\min\_\{a\\in\\Gamma\_\{t\}^\{m\}\(s\)\}\\mathcal\{R\}\_\{t\}\(s,a\)\.\(6\)The set is nonempty because a minimum\-regret action has regret zero andϵtm≥0\\epsilon\_\{t\}^\{m\}\\geq 0\.

Constructing either the calibration score in Equation \([4](https://arxiv.org/html/2608.20564#S3.E4)\) or the test\-time set in Equation \([5](https://arxiv.org/html/2608.20564#S3.E5)\) requires evaluating every admissible action from the same state\. These evaluations are counterfactual branches: they are used to calculate regret, and only the accepted branch updates the actual transcript\. Stage 3 therefore maps the proposalat,0a\_\{t,0\}to an accepted actionata\_\{t\}\. The action has not yet changed the real transcript: Stage 4 first chooses the agent who will execute it\.

#### 3\.1\.4Stage 4: Action\-Conditioned Routing and Transcript Update

After conformal acceptance choosesata\_\{t\}, the deterministic router selects speakerit=ρ⁡\(at,st\)i\_\{t\}=\\rho\(a\_\{t\},s\_\{t\}\)\. For disagreement\-focused actions,

ρ⁡\(at,st\)\\displaystyle\\rho\(a\_\{t\},s\_\{t\}\)=argmaxiJSD\(pi\(t\)∥p¯\(t\)\),\\displaystyle=\\arg\\max\_\{i\}\\operatorname\{JSD\}\\\!\\left\(p\_\{i\}^\{\(t\)\}\\parallel\\bar\{p\}^\{\(t\)\}\\right\),\(7\)at\\displaystyle a\_\{t\}∈\{Challenge,Clarify\}\.\\displaystyle\\in\\\{\\textsc\{Challenge\},\\textsc\{Clarify\}\\\}\.The most disagreeing agent is the most likely to hold evidence or an interpretation not yet incorporated into the group belief\. For evidence\-acquisition actions,

ρ⁡\(at,st\)=arg⁡mini⁡ni\(t\),at∈\{SeekEvidence,Route\},\\rho\(a\_\{t\},s\_\{t\}\)=\\arg\\min\_\{i\}n\_\{i\}^\{\(t\)\},\\\\ a\_\{t\}\\in\\\{\\textsc\{SeekEvidence\},\\textsc\{Route\}\\\},\(8\)whereni\(t\)n\_\{i\}^\{\(t\)\}is the number of messages previously contributed by agentii\. The least\-heard agent is used as a proxy for the agent most likely to retain unshared evidence\. Ties in Equations \([7](https://arxiv.org/html/2608.20564#S3.E7)\)–\([8](https://arxiv.org/html/2608.20564#S3.E8)\) are resolved by a fixed agent ordering\.

The executed control is thereforeut=\(at,it\),u\_\{t\}=\(a\_\{t\},i\_\{t\}\),where conformal communication control determinesata\_\{t\}and the router determinesiti\_\{t\}\. The selected agent generates one action\-conditioned message

mt\+1∼LLMi⁡\(x,𝒴,Ishared,Ii,ℋt,at\),m\_\{t\+1\}\\sim\\operatorname\{LLM\}\_\{i\}\\\!\\left\(x,\\mathcal\{Y\},I\_\{\\mathrm\{shared\}\},I\_\{i\},\\mathcal\{H\}\_\{t\},a\_\{t\}\\right\),which is appended to the public transcript\. All agents subsequently reportpi\(t\+1\)p\_\{i\}^\{\(t\+1\)\}, and Consilience recomputesst\+1s\_\{t\+1\}\. Thus, if no stopping condition is met, Stage 4 closes the feedback loop by returning the updated transcript to Stage 1\.

#### 3\.1\.5Stage 5: Termination and Collective Prediction

Deliberation terminates whenVoteis selected by a controller that admits it, a fixed early\-stopping condition is met, or the maximum number of roundsTTis reached\. Each agent then independently reports an answery^i∈𝒴\\widehat\{y\}\_\{i\}\\in\\mathcal\{Y\}and confidenceci∈\[0,1\]c\_\{i\}\\in\[0,1\]\. We define

N\(y\)=∑i=1N𝟏\[y^i=y\],andN\(y\)=\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\[\\widehat\{y\}\_\{i\}=y\],\\text\{ and\}
𝒴tie=\{y:N⁡\(y\)=maxy′∈𝒴⁡N⁡\(y′\)\}\.\\mathcal\{Y\}\_\{\\mathrm\{tie\}\}=\\left\\\{y:N\(y\)=\\max\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}N\(y^\{\\prime\}\)\\right\\\}\.
𝒴tie\\mathcal\{Y\}\_\{\\mathrm\{tie\}\}contains all the plurality winners\.

The final prediction is

y^final=argmaxy∈𝒴tie∑i:y^i=yci\.\\widehat\{y\}\_\{\\mathrm\{final\}\}=\\arg\\max\_\{y\\in\\mathcal\{Y\}\_\{\\mathrm\{tie\}\}\}\\sum\_\{i:\\widehat\{y\}\_\{i\}=y\}c\_\{i\}\.

### 3\.2End\-to\-End Procedure and Theoretical Guarantee

The five stages above specify one online control round:

\(ℋt,\{Ii\}i=1N\)⏟current information→Stage 1st→Stage 2at,0→Stage 3at→Stage 4ℋt\+1,\\underbrace\{\(\\mathcal\{H\}\_\{t\},\\\{I\_\{i\}\\\}\_\{i=1\}^\{N\}\)\}\_\{\\text\{current information\}\}\\xrightarrow\{\\text\{Stage 1\}\}s\_\{t\}\\xrightarrow\{\\text\{Stage 2\}\}a\_\{t,0\}\\xrightarrow\{\\text\{Stage 3\}\}a\_\{t\}\\xrightarrow\{\\text\{Stage 4\}\}\\mathcal\{H\}\_\{t\+1\},followed by either another round or the Stage\-5 prediction\. The calibration phase precedes online deliberation and supplies the round\-specific thresholds used in Stage 3\.

##### Round\-wise marginal guarantee\.

Here, we formalize the statistical guarantee for the Stage\-3 acceptance rule based on the theory of conformal prediction framework\.

Proposition[1](https://arxiv.org/html/2608.20564#Thmproposition1)specializes this construction to the round\-specific one\-step regret scores of a fixed Consilience controller and gives the sharper finite\-sample result for almost surely distinct scores\.

###### Proposition 1\(Round\-Wise Marginal Regret Coverage\)\.

Fix a controllermmand roundtt\. Letntn\_\{t\}calibration trajectories reach roundtt, with statess1,…,snts\_\{1\},\\ldots,s\_\{n\_\{t\}\}, and letstests\_\{\\mathrm\{test\}\}be the round\-ttstate of a new trajectory conditional on that trajectory reaching roundtt\. Assume that, conditional on the realized calibration\-cohort size and on the test trajectory reaching roundtt,Qtm​\(s1\),…,Qtm​\(snt\),Qtm​\(stest\)Q\_\{t\}^\{m\}\(s\_\{1\}\),\\ldots,Q\_\{t\}^\{m\}\(s\_\{n\_\{t\}\}\),Q\_\{t\}^\{m\}\(s\_\{\\mathrm\{test\}\}\)are exchangeable, and thatπm\\pi\_\{m\}and all components of the score\-generating procedure were fixed independently of the calibration sample\. Withktk\_\{t\}andϵtm\\epsilon\_\{t\}^\{m\}defined in Stage 3,

ℙ\(Qtm\(stest\)≤ϵtm\|𝒯testreachest,Nt=nt\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Q\_\{t\}^\{m\}\(s\_\{\\mathrm\{test\}\}\)\\leq\\epsilon\_\{t\}^\{m\}\\;\\middle\|\\;\\mathcal\{T\}\_\{\\mathrm\{test\}\}\\text\{ reaches \}t,\\,N\_\{t\}=n\_\{t\}\\right\)\\geq 1\-\\alpha\.\(9\)Equivalently,

ℙ\(πm\(stest\)∈Γtm\(stest\)\|𝒯testreachest,Nt=nt\)≥1−α\.\\mathbb\{P\}\\\!\\left\(\\pi\_\{m\}\(s\_\{\\mathrm\{test\}\}\)\\in\\Gamma\_\{t\}^\{m\}\(s\_\{\\mathrm\{test\}\}\)\\;\\middle\|\\;\\mathcal\{T\}\_\{\\mathrm\{test\}\}\\text\{ reaches \}t,\\,N\_\{t\}=n\_\{t\}\\right\)\\geq 1\-\\alpha\.Ifkt≤ntk\_\{t\}\\leq n\_\{t\}and the pooled calibration and test scores are almost surely distinct, the probability in Equation \([9](https://arxiv.org/html/2608.20564#S3.E9)\) equalskt/\(nt\+1\)k\_\{t\}/\(n\_\{t\}\+1\), and therefore lies in

\[1−α,1−α\+1nt\+1\)\.\\left\[1\-\\alpha,\\;1\-\\alpha\+\\frac\{1\}\{n\_\{t\}\+1\}\\right\)\.

The interpretation and scope of Proposition[1](https://arxiv.org/html/2608.20564#S3.Ex20)are in the Appendix\.

## 4Experiments

Table 1:Task and vote accuracy on HiddenBench[11](https://arxiv.org/html/2608.20564#bib.bib9)across the evaluated language models after conformal acceptance\. Each entry reports task accuracy / vote accuracy\. Hidden Pre, Hidden Post, and Full Info are reproduced from the baseline\. Higher values are better\. The best values are shown in bold\.Table 2:Task accuracy and vote accuracy for the benchmark generated from GroupTravelBench[5](https://arxiv.org/html/2608.20564#bib.bib20)averaged across 12 models\.We evaluate Consilience through four research questions:RQ1whether adaptive communication improves accuracy over no\-discussion and round\-robin baselines;RQ2whether distributed deliberation can match full\-information reasoning;RQ3how controller choice, communication actions, routing, and termination affect performance; andRQ4whether learned coordination generalizes to unseen tasks and heterogeneous agent groups\.

##### Benchmarks and models\.

We evaluate Consilience on the 65 hidden\-profile decision problems in HiddenBench\([11](https://arxiv.org/html/2608.20564#bib.bib9)\)and on a separately generated benchmark following the task structure of GroupTravelBench\([5](https://arxiv.org/html/2608.20564#bib.bib20)\)\. Each task distributes the evidence required for the correct answer across agents, which communicate only through the public transcript\. Because the original GroupTravelBench data are unavailable, our generated benchmark is not a reproduction and its results are not directly comparable; generation and validation details are in the Appendix\. We test 12 instruction\-tuned open\-weight models spanning multiple families and scales[19](https://arxiv.org/html/2608.20564#bib.bib22);[2](https://arxiv.org/html/2608.20564#bib.bib21);[1](https://arxiv.org/html/2608.20564#bib.bib23);[24](https://arxiv.org/html/2608.20564#bib.bib25);[9](https://arxiv.org/html/2608.20564#bib.bib24); model and serving details, together with details on supplementary closed\-model experiments, are also provided in the Appendix\.

### 4\.1Compared Discussion Protocols

We compare four Consilience controllers \(Rules,MLP,LLM, andLLM\+Vote\) with three references:Hidden\-Pre, in which agents vote without communicating;Hidden\-Post, which uses fixed round\-robin discussion; andFull\-Info, in which agents receive all evidence but do not discuss\.Full\-Infois an information\-rich empirical reference, not a theoretical upper bound\.

### 4\.2Evaluation Metrics

We report two complementary evaluation metrics computed from the agents’ final votes\.

Vote accuracy\.Vote accuracy measures the proportion of individual agent votes that match the ground\-truth answer:

VoteAcc=∑d∈𝒟∑i=1Nd𝕀\[vd,i=yd\]∑d∈𝒟Nd,\\mathrm\{VoteAcc\}=\\frac\{\\sum\_\{d\\in\\mathcal\{D\}\}\\sum\_\{i=1\}^\{N\_\{d\}\}\\mathbb\{I\}\[v\_\{d,i\}=y\_\{d\}\]\}\{\\sum\_\{d\\in\\mathcal\{D\}\}N\_\{d\}\},where𝒟\\mathcal\{D\}is the evaluation set,NdN\_\{d\}is the number of agents for taskdd,vd,iv\_\{d,i\}is the final vote of agentii, andydy\_\{d\}is the correct answer\.

Task accuracy\.Task accuracy measures whether the collective decision is correct\. For each task, the final prediction is obtained by plurality voting,

y^d=argmaxa∈𝒜d∑i=1Nd𝕀\[vd,i=a\],\\hat\{y\}\_\{d\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\_\{d\}\}\\sum\_\{i=1\}^\{N\_\{d\}\}\\mathbb\{I\}\[v\_\{d,i\}=a\],with ties broken using the summed confidence of supporting agents when available, and otherwise by the benchmark’s deterministic ordering\. Task accuracy is then

TaskAcc=1\|𝒟\|∑d∈𝒟𝕀\[y^d=yd\]\.\\mathrm\{TaskAcc\}=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{d\\in\\mathcal\{D\}\}\\mathbb\{I\}\[\\hat\{y\}\_\{d\}=y\_\{d\}\]\.
Task accuracy is our primary evaluation metric, while vote accuracy provides a complementary measure of how consistently individual agents converge to the correct answer\.

## 5Results and Discussion

We provide the results of two benchmarks along with their analysis in this section\. Detailed results are in the appendix\.

### 5\.1Overall Performance

Table[1](https://arxiv.org/html/2608.20564#S4.T1)reports the accuracy of each discussion protocol across the evaluated models\. The hidden pre\-discussion condition provides a lower\-information baseline in which agents cannot exchange their private evidence\. Fixed round\-robin discussion improves information availability, but does not explicitly control which evidence should be requested, which agent should provide it, or when the discussion should terminate\.

Across the evaluated models, Consilience consistently improves over these uncontrolled hidden\-information protocols\. The strongest controller obtains an average accuracy of0\.83, compared with0\.26for fixed round\-robin discussion, corresponding to an absolute improvement of0\.57\. This result supportsRQ1:the observed gains arise from adapting the communication intervention to the evolving collective state rather than from allowing agents to communicate\.

### 5\.2Can Communication Compensate for Distributed Information?

We next compare controlled hidden\-information deliberation with the full\-information, no\-discussion condition\. This comparison separates two potential sources of performance: access to evidence and the ability to integrate that evidence effectively\.

The best Consilience policy reaches or exceeds the full\-information condition on11 of the 12models\. Averaged across models, its accuracy bettered the full\-information condition by0\.13\. In almost all cases, controlled deliberation performs better despite no individual agent having direct access to all hidden facts\.

This result should not be interpreted as exceeding a theoretical upper bound\. The full\-information condition is an information\-rich empirical baseline: language models may still overlook evidence, confuse relationships among facts, or commit prematurely to an answer\. The result instead suggests that staged evidence sharing and explicit disagreement resolution can facilitate evidence integration more effectively than presenting all facts in a single context\.

### 5\.3Effect of Controller Design

All controller variants share the same actions, routing rules, agents, and prompts, differing only in action selection\. The rule\-based controller achieves the highest average accuracy0\.83, followed by the LLM0\.80, LLM\+Vote0\.80, and MLP0\.77, showing that the collective\-state variables support effective coordination even without learned control\. The two LLM controllers behave differently across model scales as seen in Tab[1](https://arxiv.org/html/2608.20564#S4.T1): adaptive voting benefits smaller models by stopping before repeated or misleading discussion causes conversational drift, whereas larger models benefit from longer deliberation, which helps uncover additional evidence without degrading context\.

### 5\.4Generalization to Unseen Tasks

To evaluate held\-out generalization, we train the MLP controller on five\-task subsets and test it on the remaining tasks; the all\-task MLP is retained only as an in\-domain oracle\-imitation diagnostic\. The unseen\-task MLP achieves0\.73average accuracy, compared with0\.77for the all\-task model, indicating that most of the learned coordination behavior transfers to unseen discussion states\. This suggests that the compact state captures recurring conditions such as disagreement, redundancy, and unsupported consensus rather than task\-specific content\. The remaining gap may reflect task\-distribution shift, variation in answer\-set size, or imperfect counterfactual reward targets\. Full training and evaluation details are provided in the appendix\.

### 5\.5Ablation Studies

We conduct five ablations that independently examine: \(1\) random sampling from conformal prediction sets, \(2\) action\-conditional speaker routing, \(3\) the contribution of each communication action, \(4\) heterogeneous agent compositions, and \(5\) controller\-state features\. In each case, all other experimental components remain fixed\. All detailed results are provided in appendix\.

#### 5\.5\.1Conformal Prediction\-Set Sampling

tests whether conformally admissible actions also support successful deliberation\. At each discussion round, we uniformly sample an action from the conformal prediction set rather than executing the controller’s preferred action, and follow the resulting trajectory until termination\. Randomly sampled actions maintain high task success across all controller variants, with only small differences between them\. Thus, the calibrated sets generally contain multiple effective actions rather than a single fragile choice, providing useful alternative deliberation paths in addition to their finite\-sample statistical guarantee\.

#### 5\.5\.2Speaker\-Routing Ablation

compares action\-conditional routing with generic policies while holding the discussion state and controller\-selected action fixed\. We measure average one\-step objective improvement and normalized regret relative to the counterfactual optimal speaker\. Action\-conditional routing achieves the largest mean objective improvement while maintaining low regret\. Random, most\-disagreeing, and most\-spoken routing produce negative mean improvements\. Round\-robin obtains marginally lower normalized regret, but substantially lower improvement, partly because near\-optimal speakers can have similar outcomes when the counterfactual performance range is small\. These results show that no single generic notion of speaker importance is sufficient\. Matching disagreement\-oriented actions to dissenting agents and evidence\-seeking actions to less\-participating agents yields more informative messages without jointly optimizing every action–speaker pair\.

#### 5\.5\.3Action Leave\-One\-Out Ablation

We remove each communication action for an entire discussion and measure the change in task accuracy\. RemovingRoutecauses the largest average drop, highlighting the importance of engaging underrepresented agents, while removingSeekNewEvidencealso substantially hurts performance, especially for LLM controllers\.Challengehas a modest effect, consistent with its targeted role in preventing premature consensus, whereasClarifyDisagreementhas the smallest effect and can slightly reduce accuracy, suggesting that routing and evidence seeking often resolve disagreement indirectly\.

#### 5\.5\.4Heterogeneous\-Agent Ablation

We replace each agent with a different language model while keeping the controller fixed, introducing variation in reasoning ability, calibration, and instruction following\. Results shows that Consilience remains effective across heterogeneous compositions\. Strong and cross\-family agent groups generally benefit most from adaptive coordination, whereas weaker compositions remain limited by their ability to extract evidence\. This suggests that Consilience can exploit complementary reasoning styles and primarily relies on observable discussion dynamics rather than model\-family\-specific interactions\.

#### 5\.5\.5Controller\-State Feature Ablation

compares MLP controllers trained with different subsets of the controller\-state representation\. The complete state achieves the highest task accuracy and lowest discussion objective\. Information\-related features \(redundancy, evidence gain, and premature consensus\) produce comparable accuracy, showing that they provide the most direct signals for action selection, but their higher objective indicates less efficient or controlled discussions\.

Removing communication cost from the combined belief\-and\-information state reduces accuracy and produces the worst objective\. Conversely, belief\-only features yield the shortest discussions but the lowest accuracy because uncertainty and disagreement do not reveal whether new evidence has been introduced or whether consensus is sufficiently supported\. The feature groups are therefore complementary: information features guide intervention selection, belief features characterize collective reasoning, and cost discourages unnecessarily long discussions\. Combining all three provides the most reliable balance between task success and discussion quality\.

## 6Conclusion

We introduced Consilience, a closed\-loop framework for adaptively selecting, certifying, and routing communication in hidden\-profile multi\-agent reasoning\. Across two benchmarks and 12 language models, Consilience consistently improves over uncontrolled discussion and often matches or exceeds full\-information baselines, demonstrating that structured communication can be as important as information access itself\. While our evaluation focuses on hidden\-profile tasks, future work will explore broader collaborative reasoning settings\.

## References

- M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1)\.
- Baiet al\.\(2023\)J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1)\.
- Chanet al\.\(2023\)C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. LiuChateval: towards better llm\-based evaluators through multi\-agent debate\.arXiv preprint arXiv:2308\.07201\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Chenet al\.\(2024\)W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\. Chan, H\. Yu, Y\. Lu, Y\. Hung, C\. Qian,et al\.Agentverse: facilitating multi\-agent collaboration and exploring emergent behaviors\.InInternational Conference on Learning Representations,Vol\.2024,pp\.20094–20136\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Chenget al\.\(2026\)X\. Cheng, Y\. Hu, L\. Zheng, Z\. Pan, X\. Li, and Y\. LiuGrouptravelbench: benchmarking llm agents on multi\-person travel planning\.arXiv preprint arXiv:2605\.25200\.Cited by:[3rd item](https://arxiv.org/html/2608.20564#S1.I1.i3.p1.1),[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2608.20564#S4.T2)\.
- Duet al\.\(2023\)Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. MordatchImproving factuality and reasoning in language models through multiagent debate\.arXiv preprint arXiv:2305\.14325\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Gigone and Hastie \(1993\)D\. Gigone and R\. HastieThe common knowledge effect: information sharing and group judgment\.\.Journal of Personality and social Psychology65\(5\),pp\.959\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p3.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, S\. Yau, Z\. Lin, L\. Zhou,et al\.MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,Vol\.2024,pp\.23247–23275\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. GhanemCamel: communicative agents for" mind" exploration of large language model society\.arXiv preprint arXiv:2303\.17760\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Liet al\.\(2025\)Y\. Li, A\. Naito, and H\. ShiradoHiddenBench: assessing collective reasoning in multi\-agent llms via hidden profile tasks\.arXiv preprint arXiv:2505\.11556\.Cited by:[3rd item](https://arxiv.org/html/2608.20564#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p4.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1),[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.20564#S4.T1)\.
- Lianget al\.\(2024\)T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. TuEncouraging divergent thinking in large language models through multi\-agent debate\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\.17889–17904\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Lin \(2002\)J\. LinDivergence measures based on the shannon entropy\.IEEE Transactions on Information theory37\(1\),pp\.145–151\.Cited by:[§3\.1\.1](https://arxiv.org/html/2608.20564#S3.SS1.SSS1.Px2.p1.2)\.
- Luet al\.\(2012\)L\. Lu, Y\. C\. Yuan, and P\. L\. McLeodTwenty\-five years of hidden profiles in group decision making: a meta\-analysis\.Personality and Social Psychology Review16\(1\),pp\.54–75\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p2.1),[§1](https://arxiv.org/html/2608.20564#S1.p3.1)\.
- Schulz\-Hardt and Mojzisch \(2012\)S\. Schulz\-Hardt and A\. MojzischHow to achieve synergy in group decision making: lessons to be learned from the hidden profile paradigm\.European Review of Social Psychology23\(1\),pp\.305–343\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p2.1),[§1](https://arxiv.org/html/2608.20564#S1.p3.1)\.
- Stasseret al\.\(1989\)G\. Stasser, L\. A\. Taylor, and C\. HannaInformation sampling in structured and unstructured discussions of three\-and six\-person groups\.\.Journal of personality and social psychology57\(1\),pp\.67\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p3.1)\.
- Stasser and Titus \(1985\)G\. Stasser and W\. TitusPooling of unshared information in group decision making: biased information sampling during discussion\.\.Journal of personality and social psychology48\(6\),pp\.1467\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p2.1),[§1](https://arxiv.org/html/2608.20564#S1.p3.1)\.
- Stasser \(1988\)G\. StasserComputer simulation as a research tool: the discuss model of group decision making\.Journal of experimental social psychology24\(5\),pp\.393–422\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p2.1)\.
- Touvronet al\.\(2023\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1)\.
- Vovket al\.\(2005\)V\. Vovk, A\. Gammerman, and G\. ShaferAlgorithmic learning in a random world\.Springer\.Cited by:[§3\.1\.3](https://arxiv.org/html/2608.20564#S3.SS1.SSS3.Px1.p1.1)\.
- Wanget al\.\(2024\)Q\. Wang, Z\. Wang, Y\. Su, H\. Tong, and Y\. SongRethinking the bounds of llm reasoning: are multi\-agent discussions the key?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.6106–6131\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1)\.
- Wuet al\.\(2024\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.Autogen: enabling next\-gen llm applications via multi\-agent conversations\.InFirst conference on language modeling,Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Xionget al\.\(2023\)K\. Xiong, X\. Ding, Y\. Cao, T\. Liu, and B\. QinExamining inter\-consistency of large language models collaboration: an in\-depth analysis via debate\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\.7572–7590\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p1.1),[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§4](https://arxiv.org/html/2608.20564#S4.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025\)K\. Zhu, H\. Du, Z\. Hong, X\. Yang, S\. Guo, D\. Z\. Wang, Z\. Wang, C\. Qian, R\. Tang, H\. Ji,et al\.Multiagentbench: evaluating the collaboration and competition of llm agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.8580–8622\.Cited by:[§1](https://arxiv.org/html/2608.20564#S1.p5.1)\.

AppendixSupplementary material: a worked hidden\-profile example, the interpretation and full proof of Proposition 1, expanded experimental results, ablations, and the complete prompt templates\.

We first provide an example of this failure mode, and how Consilience corrects it by surfacing complementary private evidence, as illustrated in Figure[2](https://arxiv.org/html/2608.20564#A0.F2)\. Then we clarify the interpretation, assumptions, and scope of the conformal coverage guarantee in Proposition 1, and provide its complete proof\. We then present additional experimental details and expanded non\-conformal results on HiddenBench and the generated travel\-planning benchmark, followed by analyses of task\-level generalization and heterogeneous\-agent groups\. Next, we report ablations of the collective\-state features, communication actions, speaker routing strategy, and conformal action\-sampling procedure\. Finally, we provide the complete prompt templates used for belief estimation, message generation, controller action selection, and message\-level evaluation, together with the generation, validation, repair, and semantic\-review prompts used to construct the synthetic benchmark\.

![Refer to caption](https://arxiv.org/html/2608.20564v1/Figures/img4.png)Figure 2:Three agents choose between Apartments A and B\. Shared evidence initially favors Apartment A, but complementary private clues collectively identify Apartment B as the correct choice\. Round\-robin discussion repeats common evidence further suppresses support for B \(as Agent 1 has no clue about any utility surcharges for Apartment B in Round 1\), and the final decision converges on A\.Consilienceinstead adaptively selects interventions and speakers to surface unshared evidence, reverse premature consensus, and recover B\.## Appendix AInterpretation, Scope, and Proof of Proposition 1

##### Coverage of the controller proposal\.

Proposition 1 provides a round\-wise marginal coverage statement for the one\-step regret of the controller’s proposed action\. LetNtN\_\{t\}denote the random number of calibration trajectories that reach discussion roundtt, and letntn\_\{t\}denote its realized value\. For a fixed controllermmand roundtt, the proposition states that

Pr⁡\(CLOSE\\displaystyle\\Pr\\\!\\Bigl\(ℛt​\(sttest,πm​\(sttest\)\)≤ϵtm\\displaystyle\\mathcal\{R\}\_\{t\}\\\!\\left\(s\_\{t\}^\{\\mathrm\{test\}\},\\pi\_\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\right\)\\leq\\epsilon\_\{t\}^\{m\}\(10\)\|𝒯testreaches roundt,Nt=nt\)≥1−α\.\\displaystyle\\bigm\|\\;\\mathcal\{T\}\_\{\\mathrm\{test\}\}\\text\{ reaches round \}t,\\,N\_\{t\}=n\_\{t\}\\Bigr\)\\geq 1\-\\alpha\.Equivalently, with probability at least1−α1\-\\alpha, the controller’s proposal belongs to the round\-specific conformal action set:

πm​\(sttest\)∈Γtm​\(sttest\)\.\\pi\_\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\in\\Gamma\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\.\(11\)
The probability in Equation \([10](https://arxiv.org/html/2608.20564#A1.E10)\) is marginal over the calibration trajectories and the new test trajectory, conditional on the test trajectory reaching roundttand onNt=ntN\_\{t\}=n\_\{t\}\. It is not, in general, a coverage guarantee conditional on one fixed realized calibration sample\.

##### Deterministic property of the executed action\.

The probabilistic coverage statement concerns the controller’s raw proposal rather than the action ultimately executed by Consilience\. The executed action is

at=\{πm​\(st\),ℛt​\(st,πm​\(st\)\)≤ϵtm,Select⁡\(Γtm​\(st\)\),ℛt​\(st,πm​\(st\)\)\>ϵtm\.a\_\{t\}=\\begin\{cases\}\\pi\_\{m\}\(s\_\{t\}\),&\\mathcal\{R\}\_\{t\}\(s\_\{t\},\\pi\_\{m\}\(s\_\{t\}\)\)\\leq\\epsilon\_\{t\}^\{m\},\\\\\[3\.0pt\] \\operatorname\{Select\}\\\!\\left\(\\Gamma\_\{t\}^\{m\}\(s\_\{t\}\)\\right\),&\\mathcal\{R\}\_\{t\}\(s\_\{t\},\\pi\_\{m\}\(s\_\{t\}\)\)\>\\epsilon\_\{t\}^\{m\}\.\\end\{cases\}\(12\)Provided that the conformal action set is evaluated using all admissible counterfactual branches and that

Select⁡\(Γtm​\(st\)\)∈Γtm​\(st\),\\operatorname\{Select\}\\\!\\left\(\\Gamma\_\{t\}^\{m\}\(s\_\{t\}\)\\right\)\\in\\Gamma\_\{t\}^\{m\}\(s\_\{t\}\),the executed action satisfies

ℛt​\(st,at\)≤ϵtm\\mathcal\{R\}\_\{t\}\(s\_\{t\},a\_\{t\}\)\\leq\\epsilon\_\{t\}^\{m\}\(13\)for every evaluated statests\_\{t\}\.

Equation \([13](https://arxiv.org/html/2608.20564#A1.E13)\) is an algorithmic consequence of the acceptance rule, rather than an additional probabilistic conformal guarantee\. If the proposal is accepted, it satisfies the threshold by definition\. If it is rejected, the fallback returns an element ofΓtm​\(st\)\\Gamma\_\{t\}^\{m\}\(s\_\{t\}\), every member of which has regret at mostϵtm\\epsilon\_\{t\}^\{m\}\. The conformal set is nonempty because at least one admissible action minimizes the realized counterfactual next\-state loss and therefore has zero one\-step regret\.

This deterministic statement assumes that the counterfactual branch used to evaluateℛt​\(st,at\)\\mathcal\{R\}\_\{t\}\(s\_\{t\},a\_\{t\}\)is the branch committed to the public trajectory\. In particular, the selected speaker message and the resulting belief updates must not be resampled after acceptance\. Otherwise, the realized next state may differ from the evaluated counterfactual state, and Equation \([13](https://arxiv.org/html/2608.20564#A1.E13)\) need not hold for the resampled outcome\.

##### Scope of the guarantee\.

The proposition bounds one\-step regret relative to the admissible action producing the lowest realized counterfactual next\-state loss under the surrogate discussion objectiveJJ\. It does not identify a unique ground\-truth communication action, guarantee that the surrogate\-optimal action improves answer correctness, establish global policy optimality, or guarantee that the final collective answer is correct\.

The proposition also does not provide simultaneous coverage over all rounds of a discussion trajectory\. In particular, the collection of round\-wise guarantees

Pr\(Qtm\(sttest\)≤ϵtm\|𝒯testreaches roundt,Nt=nt\)≥1−α\\Pr\\\!\\left\(Q\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\leq\\epsilon\_\{t\}^\{m\}\\,\\middle\|\\,\\mathcal\{T\}\_\{\\mathrm\{test\}\}\\text\{ reaches round \}t,\\,N\_\{t\}=n\_\{t\}\\right\)\\geq 1\-\\alphadoes not imply

Pr⁡\(Qtm​\(sttest\)≤ϵtm​for every reached round​t\)≥1−α\.\\Pr\\\!\\left\(Q\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\leq\\epsilon\_\{t\}^\{m\}\\text\{ for every reached round \}t\\right\)\\geq 1\-\\alpha\.A trajectory\-level guarantee would require an additional simultaneous or sequential calibration argument, which is outside the scope of Proposition 1\.

##### Round\-specific exchangeability\.

The marginal coverage statement requires exchangeability of the calibration and test proposal scores at the round under consideration\. More precisely, conditional onNt=ntN\_\{t\}=n\_\{t\}and on the test trajectory reaching roundtt, the scores

Qtm​\(st1\),…,Qtm​\(stnt\),Qtm​\(sttest\)Q\_\{t\}^\{m\}\(s\_\{t\}^\{1\}\),\\ldots,Q\_\{t\}^\{m\}\(s\_\{t\}^\{n\_\{t\}\}\),Q\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)must be exchangeable\. This condition holds under the proposition’s assumptions when calibration and test trajectories are generated by the same fixed, policy\-induced trajectory distribution and when the controller and all components of the score\-generating procedure are fixed independently of the calibration sample\.

Because communication control is sequential, exchangeability at a later round does not follow automatically from exchangeability at the initial round\. In particular, if an earlier conformal replacement changes the distribution of later test states relative to the calibration trajectories, exchangeability of the later\-round scores is an additional assumption\. The guarantee should therefore be understood as conditional on round\-specific exchangeability under the deployed policy\.

Accordingly, calibration is performed separately for every configuration for which coverage is claimed, including the controller, agent model, prompting configuration, admissible action space, routing policy, termination rule, decoding procedure, and score\-generating components\. Any learned component must be fixed before calibration or trained using data independent of the calibration sample\.

##### Finite\-sample quantile convention\.

Let

kt=⌈\(nt\+1\)​\(1−α\)⌉\.k\_\{t\}=\\left\\lceil\(n\_\{t\}\+1\)\(1\-\\alpha\)\\right\\rceil\.Whenkt≤ntk\_\{t\}\\leq n\_\{t\}, the thresholdϵtm\\epsilon\_\{t\}^\{m\}is thektk\_\{t\}\-th smallest calibration score\. Whenkt=nt\+1k\_\{t\}=n\_\{t\}\+1, including the casent=0n\_\{t\}=0, we use the standard convention

ϵtm=\+∞\.\\epsilon\_\{t\}^\{m\}=\+\\infty\.This convention yields valid but uninformative coverage because every admissible action belongs to the resulting conformal set\.

When ties occur among the pooled calibration and test scores, the nonrandomized procedure may be conservative\. The exact coverage valuekt/\(nt\+1\)k\_\{t\}/\(n\_\{t\}\+1\)stated in Proposition 1 applies whenkt≤ntk\_\{t\}\\leq n\_\{t\}and the pooled scores are almost surely distinct\.

##### Proof of Proposition 1\.

###### Proof\.

Condition on the event

ℰt=\{𝒯test​reaches round​t,Nt=nt\}\.\\mathcal\{E\}\_\{t\}=\\left\\\{\\mathcal\{T\}\_\{\\mathrm\{test\}\}\\text\{ reaches round \}t,\\,N\_\{t\}=n\_\{t\}\\right\\\}\.For notational convenience, define

Qi=Qtm\(sti\),i=1,…,nt,Q\_\{i\}=Q\_\{t\}^\{m\}\(s\_\{t\}^\{i\}\),\\qquad i=1,\\ldots,n\_\{t\},and

Qnt\+1=Qtm​\(sttest\)\.Q\_\{n\_\{t\}\+1\}=Q\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\.By assumption,

Q1,…,Qnt,Qnt\+1Q\_\{1\},\\ldots,Q\_\{n\_\{t\}\},Q\_\{n\_\{t\}\+1\}are exchangeable conditional onℰt\\mathcal\{E\}\_\{t\}\.

By the definitions ofQtmQ\_\{t\}^\{m\}andΓtm\\Gamma\_\{t\}^\{m\},

πm\(sttest\)∈Γtm\(sttest\)⟺Qnt\+1≤ϵtm\.\\pi\_\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\in\\Gamma\_\{t\}^\{m\}\(s\_\{t\}^\{\\mathrm\{test\}\}\)\\quad\\Longleftrightarrow\\quad Q\_\{n\_\{t\}\+1\}\\leq\\epsilon\_\{t\}^\{m\}\.\(14\)
First suppose thatkt=nt\+1k\_\{t\}=n\_\{t\}\+1\. By convention,ϵtm=\+∞\\epsilon\_\{t\}^\{m\}=\+\\infty, and hence

Pr⁡\(Qnt\+1≤ϵtm\|ℰt\)=1=ktnt\+1\.\\Pr\\\!\\left\(Q\_\{n\_\{t\}\+1\}\\leq\\epsilon\_\{t\}^\{m\}\\,\\middle\|\\,\\mathcal\{E\}\_\{t\}\\right\)=1=\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}\.
Now suppose thatkt≤ntk\_\{t\}\\leq n\_\{t\}, so thatϵtm\\epsilon\_\{t\}^\{m\}is thektk\_\{t\}\-th smallest calibration score\. Under conditional exchangeability, the standard split\-conformal order\-statistic argument gives

Pr⁡\(Qnt\+1≤ϵtm\|ℰt\)≥ktnt\+1\.\\Pr\\\!\\left\(Q\_\{n\_\{t\}\+1\}\\leq\\epsilon\_\{t\}^\{m\}\\,\\middle\|\\,\\mathcal\{E\}\_\{t\}\\right\)\\geq\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}\.\(15\)This inequality remains valid in the presence of ties; ties may make the nonrandomized conformal procedure conservative\.

From the definition ofktk\_\{t\},

ktnt\+1=⌈\(nt\+1\)​\(1−α\)⌉nt\+1≥1−α\.\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}=\\frac\{\\left\\lceil\(n\_\{t\}\+1\)\(1\-\\alpha\)\\right\\rceil\}\{n\_\{t\}\+1\}\\geq 1\-\\alpha\.\(16\)Combining this inequality with Equation \([15](https://arxiv.org/html/2608.20564#A1.E15)\) proves the round\-wise lower coverage bound\. Equation \([14](https://arxiv.org/html/2608.20564#A1.E14)\) then gives the equivalent action\-set statement\.

Finally, suppose thatkt≤ntk\_\{t\}\\leq n\_\{t\}and that thent\+1n\_\{t\}\+1pooled calibration and test scores are almost surely distinct\. Conditional exchangeability implies that the rank of the test score among the pooled scores is uniformly distributed over

\{1,…,nt\+1\}\.\\\{1,\\ldots,n\_\{t\}\+1\\\}\.Under distinctness,

Qnt\+1≤ϵtmQ\_\{n\_\{t\}\+1\}\\leq\\epsilon\_\{t\}^\{m\}holds if and only if the pooled rank ofQnt\+1Q\_\{n\_\{t\}\+1\}is at mostktk\_\{t\}\. Consequently,

Pr⁡\(Qnt\+1≤ϵtm\|ℰt\)=ktnt\+1\.\\Pr\\\!\\left\(Q\_\{n\_\{t\}\+1\}\\leq\\epsilon\_\{t\}^\{m\}\\,\\middle\|\\,\\mathcal\{E\}\_\{t\}\\right\)=\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}\.\(17\)Using

⌈x⌉<x\+1,\\lceil x\\rceil<x\+1,we obtain

ktnt\+1\\displaystyle\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}=⌈\(nt\+1\)​\(1−α\)⌉nt\+1\\displaystyle=\\frac\{\\left\\lceil\(n\_\{t\}\+1\)\(1\-\\alpha\)\\right\\rceil\}\{n\_\{t\}\+1\}\(18\)<1−α\+1nt\+1\.\\displaystyle<1\-\\alpha\+\\frac\{1\}\{n\_\{t\}\+1\}\.\(19\)Together with the lower bound, this proves

1−α≤ktnt\+1<1−α\+1nt\+1,1\-\\alpha\\leq\\frac\{k\_\{t\}\}\{n\_\{t\}\+1\}<1\-\\alpha\+\\frac\{1\}\{n\_\{t\}\+1\},completing the proof\. ∎

## Appendix BExperimental Details

Models\.We evaluate Consilience on 13 instruction\-tuned open\-weight language models spanning multiple families and scales, including Qwen3 \(0\.6B, 1\.7B, 4B, 8B, 14B, 32B, and 235B\), Qwen2\.5\-32B, Mistral\-24B, Phi\-4, Llama\-3\.1\-8B, Llama\-3\.3\-70B, and Gemma\-3\-27B\. We further include GPT\-4\.1\-mini and Claude\-3\-Haiku in a supplementary closed\-model experiment comparing the full\-information baseline against LLM\-based orchestration with adaptive voting to assess whether the observed trends generalize beyond open\-weight models\. Unless otherwise specified, all agents within a discussion instantiate the same underlying language model\.

For LLM\-based controller, the same model additionally serves as the controller responsible for selecting communication actions, whereas the learned\-controller variants replace this decision process with a lightweight MLP while retaining the evaluated LLM for belief estimation, message generation, transcript analysis, and final voting\.

## Appendix CNon\-Conformal Results

Table 3:Task and vote accuracy for Hiddenbench across the evaluated language models\. Each entry reports*task accuracy / vote accuracy*\. Higher values are better\.Table 4:Task and vote accuracy for the generated benchmark across the evaluated language models\. Each entry reports*task accuracy / vote accuracy*\. Higher values are better\.##### HiddenBench\.

Table[3](https://arxiv.org/html/2608.20564#A3.T3)shows that access to distributed evidence alone is insufficient without effective coordination\. Averaged across the 15 models, hidden pre\-discussion achieves only0\.0830\.083task accuracy, while uncontrolled round\-robin discussion improves this to0\.2850\.285\. Providing all evidence directly increases average accuracy to0\.7310\.731, confirming that the main difficulty arises from recovering and integrating information distributed across agents\. All four Consilience controllers exceed the full\-information reference on average: Rules achieves0\.8430\.843task accuracy, followed by LLM at0\.8160\.816, LLM\+Vote at0\.8150\.815, and MLP at0\.7850\.785\. The rule\-based controller therefore provides the strongest average performance, although the best controller varies across models\. In particular, the LLM\-based controllers are competitive for stronger models, and adaptive voting produces the best results for models such as Qwen3\-14B and Phi\-4\. Task and vote accuracy are also closely aligned for all controllers, indicating that improvements in the final collective answer generally correspond to broader agreement among the individual agents rather than being produced only by plurality aggregation\.

Supplementary experiments with frontier API models show the same trend: Consilience with LLM\+Vote matches or exceeds full\-information reasoning, achieving0\.9380\.938versus0\.8770\.877on openai’s gpt models and0\.8620\.862versus0\.6460\.646on Claude\-3\-Haiku\.

##### Generated benchmark\.

Table[4](https://arxiv.org/html/2608.20564#A3.T4)presents a different performance profile\. Hidden pre\-discussion obtains an average task accuracy of0\.2540\.254, while uncontrolled discussion increases it to0\.4980\.498\. Interestingly, hidden post\-discussion outperforms the full\-information condition, which achieves only0\.3660\.366\. This suggests that merely placing all evidence in one context does not guarantee that the model will identify and combine the relevant facts; interaction can help expose, repeat, and reconcile evidence that may otherwise be overlooked\. Among the controlled methods, LLM\+Vote achieves the highest average task accuracy at0\.6680\.668, followed closely by Rules at0\.6540\.654, LLM at0\.6150\.615, and MLP at0\.5070\.507\. Relative to uncontrolled discussion, LLM\+Vote improves task accuracy by0\.1700\.170, whereas the MLP improves it by only0\.0090\.009\. Adaptive termination is therefore especially useful on this benchmark, where continued discussion may reinforce an incorrect answer after the relevant evidence has already been surfaced\.

Performance on the generated benchmark also depends strongly on the capability of the participating model\. The smallest Qwen3 models receive limited benefit from communication control, whereas Qwen2\.5\-32B and Llama\-3\.3\-70B exceed0\.880\.88task accuracy under the strongest controllers\. The MLP controller is less consistent under this setting, performing substantially below Rules and the LLM\-based controllers for models such as Mistral\-24B and Qwen3\-32B\. This pattern suggests that the fixed learned mapping from collective\-state features to actions is more sensitive to changes in benchmark structure or discussion dynamics, while an LLM controller can adapt its action choice using the current transcript and beliefs\.

##### Comparison across benchmarks\.

Across the 12 models shared by both tables, controller performance is generally lower on the generated benchmark\. Relative to HiddenBench, average task accuracy decreases by0\.1600\.160points for Rules,0\.2610\.261points for MLP, and0\.1700\.170points for LLM, but by only0\.1150\.115points for LLM\+Vote\. Thus, LLM\+Vote exhibits the smallest cross\-benchmark degradation, while the MLP exhibits the largest\. Together, these results indicate that explicit communication control consistently improves hidden\-profile reasoning, but the most effective controller depends on the discussion distribution: simple state\-dependent rules are particularly effective on HiddenBench, whereas transcript\-aware control with adaptive voting transfers more robustly to the generated benchmark\.

### C\.1Generalization to Held\-Out Tasks

The standard MLP controller is trained using exhaustive action outcomes collected from all benchmark tasks\. It therefore measures how accurately a lightweight learned controller can imitate the one\-step action oracle within the task distribution used to generate its supervision\. However, because the same tasks contribute to both controller training and downstream evaluation, this condition should be interpreted primarily as an in\-domain oracle\-imitation diagnostic rather than as a strict test of task\-level generalization\.

We therefore construct a separate held\-out evaluation\. HiddenBench is partitioned into a five\-task training subset and a disjoint 60\-task test set\. For each training task, the exhaustive action explorer follows the current discussion trajectory and, at every reached state, executes each of the four available communication actions:Challenge,ClarifyDisagreement,SeekNewEvidence, andRoute\.

The resulting training examples pair the six\-dimensional collective\-state representation with a four\-dimensional vector containing the observed reward of every communication action\. The MLP is trained only on examples generated from the five training tasks\. Its feature\-normalization statistics, reward\-normalization statistics, and network parameters are therefore estimated exclusively from the training split\. After training, the controller is frozen and evaluated on the remaining 60 tasks without any additional fitting or access to their counterfactual action outcomes\. At each test\-time discussion state, the MLP predicts the expected immediate reward of the four actions and executes the action with the highest predicted value\. Agent prompting, speaker routing, belief estimation, and final voting remain unchanged\.

This evaluation measures whether the mapping learned from abstract collective\-state variables to communication actions transfers across decision problems\. The controller does not observe task identifiers or raw private clues directly; it acts on quantities such as group\-belief entropy, inter\-agent disagreement, message redundancy, communication cost, evidence gain, and premature consensus\. Consequently, successful transfer would indicate that these variables capture recurring coordination conditions rather than merely memorizing the content or trajectory of individual benchmark tasks\.

Across the 13 models for which held\-out results are available, the held\-out MLP obtains an average task accuracy of0\.73\. The corresponding all\-task MLP average is0\.77\. Thus, withholding the evaluation tasks during controller training is associated with a descriptive decrease of approximately 0\.04\. The relatively small aggregate gap suggests that a substantial portion of the learned action\-selection behavior transfers to discussion trajectories that were not observed during training\.

Transfer is particularly strong for medium and large language models\. 8 of the 13 evaluated models achieve at least 0\.83 held\-out task accuracy\. Qwen3\-32B, Qwen3\-235B, Llama\-3\.3\-70B, and Mistral\-24B each reach 0\.86 task accuracy, while Qwen3\-8B, Qwen3\-14B, Qwen2\.5\-32B, and Gemma\-3\-27B reach 0\.83\. These results indicate that the controller trained on only a small subset of tasks can still select useful communication actions when paired with agents capable of producing stable beliefs and informative responses\.

The largest degradations relative to the all\-task MLP occur for Qwen3\-1\.7B and Phi\-4, whose task accuracies decrease by 0\.17 and 0\.16 respectively\. Qwen3\-0\.6B also decreases by 0\.09 points\. This pattern suggests that held\-out transfer is less reliable when the participating agents produce noisier belief distributions or when their discussion states differ substantially from those represented in the controller\-training trajectories\. Because the MLP observes only the collective\-state summary, changes in the quality or calibration of agent beliefs can produce a state distribution that is difficult to interpret using a controller trained from a small number of tasks\.

Several models show comparable or slightly higher accuracy under the held\-out controller\. For example, held\-out task accuracy increases by 0\.06 points for Qwen3\-14B, 0\.07 points for Llama\-3\.1\-8B, and 0\.03 points for both Qwen3\-32B and Mistral\-24B\. These increases should not be interpreted as evidence that using less training data improves the controller\. The two conditions are evaluated over different task sets, and the deliberations themselves involve stochastic language\-model outputs\. Small positive differences can therefore arise from variation in task difficulty, generated messages, belief estimates, or action trajectories\.

Task and vote accuracy remain closely aligned under held\-out evaluation\. Their mean absolute difference across models is approximately 0\.02 accuracy points, and for Qwen3\-8B, Qwen3\-14B, and Llama\-3\.3\-70B the two values are identical\. This agreement indicates that held\-out performance is generally supported by the individual agents’ final judgments rather than arising only from plurality aggregation or confidence\-based tie\-breaking\. Larger discrepancies for a few models nevertheless show that communication control can affect both the correctness of the group decision and the distribution of support among agents\.

Overall, the held\-out results provide evidence that the controller learns reusable coordination behavior from the collective\-state representation\. The performance decrease relative to the in\-domain MLP is modest on average, and high accuracy is retained across most medium and large models despite training on only five tasks\. At the same time, the larger losses for some weaker or behaviorally distinct models show that transfer is not uniform\. The learned controller generalizes most reliably when the test\-time discussion dynamics resemble the state distributions encountered during training\.

For a strictly controlled estimate of the generalization gap, the all\-task MLP should additionally be evaluated on the same 60 held\-out tasks\. The current comparison uses the 65\-task result for the standard MLP and the 60\-task result for the held\-out MLP; consequently, the reported 0\.03 and 0\.04 point differences combine controller\-training effects with a small difference in the evaluation task set\. Evaluating both checkpoints on the identical 60\-task test split would isolate the effect of withholding task\-level training supervision\.

## Appendix DHeterogeneous\-Agent Analysis

Table[5](https://arxiv.org/html/2608.20564#A4.T5)evaluates whether Consilience remains effective when the participating agents use different underlying language models\. Overall, heterogeneous groups can achieve performance comparable to strong homogeneous groups, but their results are substantially more sensitive to model composition, agent assignment, and controller choice\. Among the heterogeneous four\-agent configurations, the rule\-based controller reaches its highest task accuracy of0\.9660\.966on both Position Rot\-4 and Ladder L2, while the LLM and LLM\+Voting controllers reach0\.9480\.948on Ladder L4\. These results show that model heterogeneity does not inherently prevent successful coordination when the group contains sufficiently capable and complementary agents\.

The position\-rotation experiments reveal that assigning the same four models to different agent positions can materially alter performance\. For example, rule\-based accuracy varies from0\.8790\.879in the documented assignment to0\.9660\.966in Position Rot\-4, while LLM\+Voting varies from0\.8280\.828to0\.9310\.931across the rotations\. Thus, performance depends not only on which models are present, but also on which private evidence each model receives\. Using the stronger Qwen3\-32B controller further improves the documented heterogeneous group, raising MLP accuracy from0\.7760\.776to0\.9310\.931, LLM accuracy from0\.8450\.845to0\.8970\.897, and LLM\+Voting accuracy from0\.8790\.879to0\.9310\.931\. This indicates that controller capability can compensate for some of the variability introduced by heterogeneous participants\.

Group composition also has a clear effect\. Configurations composed primarily of capable models, such as Family\-Qwen, Spread\-Low, and Ladder L4, remain strong across several controllers\. In contrast, groups containing several small models perform substantially worse: Star and Ladder L1 obtain only0\.5860\.586task accuracy under LLM\+Voting\. The Weak\-Link configuration shows that introducing a single weak agent does not necessarily cause failure, particularly for Rules, but can reduce the effectiveness of adaptive voting\. Among homogeneous groups, Qwen3\-32B with LLM\+Voting achieves the highest overall task accuracy of0\.9830\.983, while the other strong homogeneous groups also remain consistently competitive\. Taken together, the results suggest that Consilience generalizes to heterogeneous teams, but reliable performance depends on both the capability distribution within the group and the controller’s ability to route communication across differently capable agents\.

Table 5:Performance of heterogeneous and homogeneous multi\-agent groups on the corresponding HiddenBench agent\-count partitions\. Four\-agent configurations are evaluated on the 58\-task four\-agent partition, while Documented\-3Agent is evaluated on the seven\-task three\-agent partition\. Each controller entry reports*task accuracy / vote accuracy*\. The orchestration controller is Qwen3\-8B unless marked with†\\dagger, where Qwen3\-32B is used\. Model abbreviations are: Q0\.6, Q1\.7, Q4, Q8, Q14, and Q32 for Qwen3 at the corresponding parameter scale; Q2\.5\-32 for Qwen2\.5\-32B; P4 for Phi\-4; M24 for Mistral\-24B; G27 for Gemma\-3\-27B; L8 for Llama\-3\.1\-8B; and L70 for Llama\-3\.3\-70B\.
## Appendix EAblations

##### State\-feature ablation\.

Table[6](https://arxiv.org/html/2608.20564#A5.T6)evaluates how different subsets of the collective\-state representation affect the learned MLP controller\. Using the complete feature set achieves the best overall trade\-off, with 82\.22% task success and the lowest final objective value of 0\.0239\. The information\-only controller matches this success rate, but incurs slightly higher communication cost and a substantially worse final objective, indicating that the remaining state variables improve the quality and efficiency of the learned policy even when they do not change the final task accuracy\.

Removing information\-related features reduces performance\. Both the belief\-plus\-information and belief\-plus\-cost variants achieve 77\.78% success, while the belief\-only controller performs worst at 75\.56%\. Although the belief\-only representation produces the lowest communication cost, this reduction is accompanied by a 6\.66 percentage\-point drop in success relative to the full model\. Among the reduced representations, belief plus cost yields a lower final objective than belief plus information, suggesting that explicit cost awareness helps the controller avoid inefficient deliberation\. Overall, the results show that belief statistics alone are insufficient, while combining belief, information, and cost\-related signals produces the most balanced controller\.

Table 6:State\-feature ablation for Qwen3\-8B\. Each row retrains the MLP using only the retained feature groups\. Higher success is better; lower communication cost and final objective are better\.
##### Action ablation\.

Table[7](https://arxiv.org/html/2608.20564#A5.T7)evaluates the contribution of each communication action by removing it while keeping the remaining controller unchanged\. Across controllers, routing and new\-evidence solicitation are the most consequential components\. Removing routing produces the largest average drop, reducing task accuracy by 4\.6 percentage points, while removingSeekNewEvidencedecreases average accuracy by 3\.8 points\. These effects are especially pronounced for the LLM\+Voting controller, where removing routing and new\-evidence solicitation reduces accuracy by 13\.8 and 12\.3 points, respectively\. This suggests that adaptive termination is particularly dependent on directing the discussion toward underrepresented agents and unshared evidence before voting occurs\.

The effects vary across controller types\. The rule\-based controller depends most strongly on routing, whereas the standard LLM controller is most affected by removingSeekNewEvidence\. RemovingChallengecauses a small but consistent degradation for Rules, MLP, and LLM, indicating that explicitly testing the leading hypothesis provides a modest benefit\. In contrast, removingClarifyDisagreementhas no average effect, and removing voting leaves the Rules and LLM\+Voting results unchanged in this ablation\. Several removals improve performance, including the removal ofSeekNewEvidencefrom the MLP controller, showing that individual actions are not uniformly useful and may interact with the controller’s selection policy\. Overall, the results indicate that the full action set is most valuable for providing complementary interventions, with routing and new\-evidence acquisition contributing the strongest aggregate gains\.

Table 7:Action ablation study on HiddenBench\. Each row removes one communication action from the full controller\. Entries report task accuracy \(%\), with the change relative to the corresponding full action set shown in parentheses\. Negative values indicate performance degradation\.
##### Speaker\-routing ablation\.

Table[8](https://arxiv.org/html/2608.20564#A5.T8)isolates the effect of speaker selection by holding the controller\-selected communication action fixed\. Consilience’s action\-conditional router achieves the lowest mean objective change,Δ​J=−0\.0079\\Delta J=\-0\.0079, indicating that its selected speakers produce the strongest average immediate improvement in the discussion state\. Round\-robin is the closest alternative, withΔ​J=−0\.0017\\Delta J=\-0\.0017, while random routing slightly worsens the objective on average \(Δ​J=0\.0145\\Delta J=0\.0145\)\.

The normalized\-regret results are less decisive\. Round\-robin obtains the lowest regret at0\.48390\.4839, followed closely by action\-conditional routing at0\.48680\.4868and random routing at0\.49300\.4930\. Thus, although the proposed router performs best under the mean objective\-change metric, its advantage over simple round\-robin and random selection is small, and it does not achieve the lowest normalized regret\. In contrast, selecting the most\-disagreeing or most\-spoken agent performs substantially worse under both metrics, suggesting that disagreement or participation frequency alone is insufficient for identifying the most useful speaker\. Overall, the results support action\-aware routing over these stronger heuristic alternatives, but provide only limited evidence of a clear advantage over round\-robin routing\.

##### Conformal\-sampling ablation\.

Table[9](https://arxiv.org/html/2608.20564#A5.T9)compares the four controller variants over 400 trials with Qwen3\-8B\. The LLM and MLP controllers achieve the highest mean success rate, both reaching 82\.75%, despite using different action\-selection mechanisms\. Their discussion lengths are also similar, averaging 9\.54 and 9\.87 rounds per trial, respectively\. This indicates that the lightweight learned controller can match the success of the LLM controller, although it requires slightly more communication\.

Table 8:Counterfactual speaker\-routing ablation with Qwen3\-8B\. The controller\-selected action is held fixed and only the speaker\-selection policy is changed\. Lower objective change is better, while lower normalized regret indicates routing decisions closer to the counterfactual optimum\.The rule\-based controller terminates substantially earlier, after only 3\.24 rounds on average, but its success decreases to 64\.50%\. The LLM\+Voting variant also conducts shorter discussions, averaging 5\.43 rounds, yet attains the lowest success rate of 63\.00%\. These results reveal a clear trade\-off between communication length and task performance: controllers that allow longer deliberation are substantially more successful, whereas aggressive or adaptive termination can stop the discussion before complementary private evidence has been sufficiently integrated\. Overall, conformal sampling preserves comparable performance for the LLM and MLP controllers, while the weaker results of the early\-terminating variants suggest that calibrated action selection alone does not compensate for premature voting\.

Table 9:Comparison of controller methods over 400 trials\. Using Qwen3\-8B model\.

## Appendix FComplete Prompt Templates

This section reports the complete prompt templates used in our experiments\. Text enclosed in angle brackets denotes a runtime placeholder populated separately for each task or deliberation state\.

1Youare<AGENT\_NAME\>,oneparticipantinamulti\-agentgroupdecisiontask\.

2

3Taskdescription:

4<TASK\_DESCRIPTION\>

5

6Possibleanswers:

7<POSSIBLE\_ANSWERS\>

8

9Sharedinformationvisibletoeveryone:

10\-<SHARED\_INFORMATION\_ITEM\_1\>

11\-<SHARED\_INFORMATION\_ITEM\_2\>

12\-…

13

14Informationvisibleonlytoyou:

15\-<PRIVATE\_INFORMATION\_ITEM\_1\>

16\-<PRIVATE\_INFORMATION\_ITEM\_2\>

17\-…

18

19Rules:

20\-Youmayusetheinformationvisibleonlytoyou\.

21\-Sharedecision\-relevantevidencewhenithelpsthegroup\.

22\-Donotinventfacts\.

23\-Beconcise\.

24

25Publictranscriptsofar:

26\(emptytranscript\)

27

28Giveyourcurrentbeliefoverthepossibleanswers\.ReturnJSONonly:

29\{

30"belief":\{"OPTION":0\.5\},

31"best\_answer":"oneoptionexactlyaswritten",

32"confidence":0\.0,

33"one\_sentence\_evidence":"shortevidencesummary",

34"needs\_more\_information":true

35\}

36

37Includeeverypossibleanswerexactlyandmaketheprobabilitiessumto1\.

38Possibleanswers:<POSSIBLE\_ANSWERS\>

1Youare<AGENT\_NAME\>,oneparticipantinamulti\-agentgroupdecisiontask\.

2

3Taskdescription:

4<TASK\_DESCRIPTION\>

5

6Possibleanswers:

7<POSSIBLE\_ANSWERS\>

8

9Sharedinformationvisibletoeveryone:

10\-<SHARED\_INFORMATION\_ITEM\_1\>

11\-<SHARED\_INFORMATION\_ITEM\_2\>

12\-…

13

14Informationvisibleonlytoyou:

15\-<PRIVATE\_INFORMATION\_ITEM\_1\>

16\-<PRIVATE\_INFORMATION\_ITEM\_2\>

17\-…

18

19Rules:

20\-Youmayusetheinformationvisibleonlytoyou\.

21\-Sharedecision\-relevantevidencewhenithelpsthegroup\.

22\-Donotinventfacts\.

23\-Beconcise\.

24

25Publictranscriptsofar:

26<PUBLIC\_TRANSCRIPT\>

27

28Controllerinstruction:

29<ACTION\_SPECIFIC\_INSTRUCTION\>

30

31Writeyournextmessagetothegroup\.Sharerelevantevidence,clearlystate

32ifitrulesoutanoption,communicatenaturally,anddonotrefertoitas

33privateinformation\.

1Thecurrentleadingoptionis<LEADING\_OPTION\>\.Stateanyevidencethatcould

2supportorchallenge<LEADING\_OPTION\>\.Ifyourinformationrulesoutanother

3option,saysoclearly\.

1Yourbeliefdiffersfromthegroup\.Explaintheevidencebehindyourview,

2especiallyanyfactthatrulesoutanoption\.

1Shareonenewdecision\-relevantfact\.Focusonfactsthateliminateorsupportanoption\.Avoidrepeatingthetranscript\.

1Contributeoneconcisepieceofdecision\-relevantevidence\.Ifpossible,

2explainwhichoptionitsupportsorrulesout\.

1Youaretheconsiliencecontroller\.Choosethenextcontrolleractionfora

2multi\-agentdecisiontask\.Youmustchooseexactlyoneexistingactiontype;donot

3chooseaspeakeranddonotwritetheagent’smessage\.

4

5Availableactions:

6\-challenge:askanagenttotestthecurrentleadingoptionandsurfacecontraryevidence\.

7\-clarify\_disagreement:asktheagentwhosebeliefdiffersmostfromthegrouptoexplainwhy\.

8\-seek\_new\_evidence:asktheleast\-heardagentforonenewdecision\-relevantfact\.

9\-route:asktheleast\-heardagentforconciseevidencethatsupportsorrulesoutanoption\.

10

11Task:

12<TASK\_DESCRIPTION\>

13

14Possibleanswers:

15<POSSIBLE\_ANSWERS\>

16

17Publictranscriptsofar:

18<PUBLIC\_TRANSCRIPT\>

19

20Currentstate:

21<COLLECTIVE\_STATE\_JSON\>

22

23Currentround:

24<CURRENT\_ROUND\>

25

26Currentagentbeliefs:

27<AGENT\_BELIEFS\_JSON\>

28

29Timeseachagenthasspoken:

30<SPEAKER\_COUNTS\_JSON\>

31

32ReturnJSONonly:

33\{

34"action":"oneof:challenge,clarify\_disagreement,seek\_new\_evidence,route",

35"reason":"briefreasonforchoosingthisaction",

36"expected\_effect":"briefdescriptionofwhatthisshouldimprove"

37\}

1Youaretheconsiliencecontroller\.Choosethenextcontrolleractionfora

2multi\-agentdecisiontask\.Youmustchooseexactlyoneexistingactiontype;donot

3chooseaspeakeranddonotwritetheagent’smessage\.

4

5Availableactions:

6\-challenge:askanagenttotestthecurrentleadingoptionandsurfacecontraryevidence\.

7\-clarify\_disagreement:asktheagentwhosebeliefdiffersmostfromthegrouptoexplainwhy\.

8\-seek\_new\_evidence:asktheleast\-heardagentforonenewdecision\-relevantfact\.

9\-route:asktheleast\-heardagentforconciseevidencethatsupportsorrulesoutanoption\.

10\-vote:endthediscussionnowandcollecteveryagent’sfinalvote\.

11

12Task:

13<TASK\_DESCRIPTION\>

14

15Possibleanswers:

16<POSSIBLE\_ANSWERS\>

17

18Publictranscriptsofar:

19<PUBLIC\_TRANSCRIPT\>

20

21Currentstate:

22<COLLECTIVE\_STATE\_JSON\>

23

24Currentround:

25<CURRENT\_ROUND\>

26

27Currentagentbeliefs:

28<AGENT\_BELIEFS\_JSON\>

29

30Timeseachagenthasspoken:

31<SPEAKER\_COUNTS\_JSON\>

32

33ReturnJSONonly:

34\{

35"action":"oneof:challenge,clarify\_disagreement,seek\_new\_evidence,route,vote",

36"reason":"briefreasonforchoosingthisaction",

37"expected\_effect":"briefdescriptionofwhatthisshouldimprove"

38\}

1Analyzeamessageinamulti\-agentdecisionconversation\.

2

3Options:<POSSIBLE\_ANSWERS\>

4Transcriptbeforethismessage:

5<PUBLIC\_TRANSCRIPT\_BEFORE\_MESSAGE\>

6

7Newmessage:

8<NEW\_MESSAGE\>

9

10ReturnJSONonly:

11\{

12"new\_evidence\_score":0\.0,

13"redundancy\_score":0\.0,

14"supports\_options":\["optionnamesexactly"\],

15"attacks\_or\_rules\_out\_options":\["optionnamesexactly"\],

16"is\_clarification":false,

17"short\_explanation":"onesentence"

18\}

19

20Usescoresbetween0and1\.Newevidenceaddstask\-relevantinformation;redundancy

21meansthemessagemostlyrepeatstheearliertranscript\.

### F\.1Synthetic Benchmark Generation Prompts

The additional travel\-planning benchmark was generated using structured prompting followed by deterministic validation, automatic repair, and LLM\-based semantic quality review\. Text enclosed in angle brackets denotes a value populated at runtime\.

1Youdesignadversarial,realisticmulti\-agenttravel\-planning

2benchmarks\.ReturnexactlyonevalidJSONobjectandnomarkdown\.Neverrevealorhintatthe

3correctansweroutsidethedeclaredcorrect\_answerfield\.Useonlytheschemaandoperators

4specifiedbytheuser\.

1CreateoneHiddenBench\-stylegrouptraveldecisiontask\.

2

3Variationseed:<VARIATION\_SEED\>

4Datasetposition:<TASK\_INDEX\>

5Avoidduplicatingtheseacceptedtasksignatures:<ACCEPTED\_TASK\_SIGNATURES\>

6Useexactly<NUMBER\_OF\_TRAVELERS\>travelers\.

7Makeoption<TARGET\_CORRECT\_OPTION\>thedeclaredcorrectanswer\.

8UseHiddenBench\-styleweightedpreferenceaggregation:

9\-satisfiedmust=\+2

10\-satisfiedprefer=\+1

11\-matchedavoid=\-1

12\-matchedreject=\-2

13Thedeclaredanswermustbetheuniquehighest\-scoringoption\.Itmayviolateastrongpreference

14ifitremainsthebestgroupcompromise\.Distractorsdonotneedartificial,uniqueviolation

15patterns\.

16

17Thetaskmust:

18\-involveexactly3or4travelerswithplausiblerolesandaconcretegroup\-tripcontext;

19\-exposeexactlyfourfullyspecifiedcandidateplans\(A,B,C,D\)inpublicinformation;

20\-distributeprivatestructuredpreferencesacrosstravelersusingalltiers:must,prefer,

21avoid,reject;

22\-makethedeclaredansweruniquelybestunderthevalidationprofileabove;

23\-requireevidencefrommultipletravelerstoidentifytheanswer;

24\-remainsolvablebyliteralcomparisononly:nooutsidetravelfacts,arithmeticambiguity,

25subjectiveinterpretation,compromise,orunstatedassumptions;

26\-makeeverycandidatefeasibleinordinaryreal\-worldterms;

27\-giveeverytraveleratleastonepreferencethatmatchessomebutnotallcandidates;

28\-avoidunsafe,discriminatory,medicallyimplausible,orinternallycontradictorycontent\.

29

30Tiersemantics:

31\-must:matchingisastrongpositivepreferenceworth\+2\.

32\-reject:matchingisastrongnegativepreferenceworth\-2\.

33\-prefer:matchingadds\+1utility\.

34\-avoid:matchingsubtracts1utility\.

35

36Allowedoperators:eq,neq,in,not\_in,contains,contains\_all,contains\_any,excludes,lte,gte\.

37Forcontains/excludesoperators,candidateattributesmustbearrays\.‘in‘meansascalarorany

38arrayitembelongstothepreferencevaluelist;‘not\_in‘meansnoscalar/arrayitembelongsto

39thatlist\.Forlte/gte,attributesmustbenumbers\.Everypreferencemustreferenceanattribute

40presentineveryoption\.

41

42Everyoptionattributesobjectmustincludeatleasttheseexactkeys:

43transport\_mode,departure\_window,duration\_days,price\_per\_person\_usd,hotel\_category,

44room\_arrangement,step\_free,daily\_active\_hours,max\_walk\_km\_per\_day,meal\_options,activities\.

45Youmayaddasmallnumberofconcretekeyssuchasarrival\_time,layovers,orneighborhood\.

46Allfouroptionsmustusethesamekeysandcomparableunits\.

47

48Returnthisexactshape:

49\{

50"name":"short\_unique\_snake\_case\_name",

51"description":"sharedscenario,datesorseason,origin,destination,grouprelationship",

52"public\_information":\[

53"2\-5publicfactsthatdonotencodeanypreferenceoranswer"

54\],

55"options":\[

56\{

57"id":"A",

58"title":"conciseplanlabel",

59"attributes":\{

60"transport\_mode":"train",

61"departure\_window":"morning",

62"duration\_days":4,

63"price\_per\_person\_usd":900,

64"hotel\_category":"midrange",

65"room\_arrangement":"twotwinrooms",

66"step\_free":true,

67"daily\_active\_hours":7,

68"max\_walk\_km\_per\_day":5,

69"meal\_options":\["vegetarian","seafood"\],

70"activities":\["museum","market"\]

71\}

72\}

73\],

74"agents":\[

75\{

76"id":"traveler\_1",

77"role":"realisticroleinthisgroup",

78"context":"onesentenceexplainingwhythepreferencesareplausible",

79"preferences":\[

80\{

81"id":"t1\_must\_1",

82"tier":"must",

83"attribute":"step\_free",

84"operator":"eq",

85"value":true,

86"statement":"Anaturalfirst\-personstatementwithclearnon\-negotiableforce\."

87\},

88\{

89"id":"t1\_reject\_1",

90"tier":"reject",

91"attribute":"transport\_mode",

92"operator":"eq",

93"value":"flight",

94"statement":"Anaturalfirst\-personstatementclearlyrefusingflights\."

95\}

96\]

97\}

98\],

99"correct\_answer":"oneofA/B/C/D"

100\}

101

102Use2\-4preferencespertraveler\.Individualtravelersmayhaveonlyhardoronlysoftitems,but

103thecompletetaskmustuseeachofmust,prefer,avoid,andreject\.NEVERomitthe"tier"key:

104copyitslowercasevaluefromthepreferenceid\.Beforeresponding,evaluateeverypredicate

105againstA/B/C/D,calculatetheaggregatescoreforalloptions,andverifythateachtravelerhas

106atleastonepredicatematching1\-3options\.Itisacceptableforanindividualtravelertorank

107thedeclaredanswerfirst;atleasttwoprofilesmuststillcontributenon\-constantevidenceto

108theaggregatedecision\.Thestatementmustfaithfullyexpressitspredicateandtier\.Donotadd

109evaluation,rationale,violationsummaries,orextratop\-levelfields\.

1Apreviousblueprintwasabandonedafterfailedrepairs\.Createagenuinelynewblueprintand

2avoidthesefailures:

3<PRIOR\_VALIDATION\_ISSUES\>

### F\.2Deterministic Validation and Repair

Generated tasks are first evaluated by a deterministic validator\. Under the HiddenBench validation profile, a matched must contributes\+2\+2, a matched prefer contributes\+1\+1, a matched avoid contributes−1\-1, and a matched reject contributes−2\-2\. The declared answer must be the unique highest\-scoring candidate\. Instances that fail validation are repaired using the following prompt\.

1RepairtherejectedHiddenBenchtravelblueprintbelowinplace\.

2

3Datasetposition:<TASK\_INDEX\>

4Donotduplicateacceptedsignatures:<ACCEPTED\_TASK\_SIGNATURES\>

5

6Changetheminimumoptionattributesorpreferencesneededtofixeveryvalidatorissue\.Preserve

7thescenario,travelerroles,fouroptionids,andallalready\-validcontent\.Returnthecomplete

8repairedblueprintasoneJSONobject,notapatch\.

9

10Foran"expectedexactlyonehard\-feasibleoption,found\[\]"errorinstrictmode,repairinthisorder:

111\.Readthedetailedviolationlistforthedeclaredcorrect\_answer\.

122\.Changeitspublicattributes,ortheconflictingpreferencepredicateandmatchingstatement,

13untilitviolateszeroMUST/REJECTpreferences\.

143\.Re\-evaluateeveryhardpredicateliterallyagainstallfouroptions\.

154\.Ensureeachotheroptionretainsadifferentnon\-emptyviolationset\.

16Donotmerelyrepeattherejectedblueprint\.

17

18Requiredinvariantsafterrepair:

19\-exactlyfourcomparableplansandexactly3or4travelers;

20\-everypreferencecontainsid,tier,attribute,operator,value,andstatement;

21\-tierisexactlyonelowercasevalue:must,prefer,avoid,orreject;

22\-eachtravelerhasatleasttwopreferencesandatleastonepredicatethatmatchessomebutnot

23allplans;

24\-thetaskcollectivelyusesallfourtiers;

25\-usingmust=\+2,prefer=\+1,avoid=\-1,reject=\-2,correct\_answeristheunique

26highest\-scoringoption;

27\-anindividualtravelermayrankcorrect\_answerfirst,butatleasttwoprofilesmustcontribute

28non\-constantoptionscores;

29\-plansmayviolateMUSTorREJECTitems;donotforceafullyhard\-feasibleplan;

30\-everypreferencestatementexactlymatchesitsstructuredpredicate\.

31

32Validatorissues:

33<VALIDATOR\_ISSUES\_JSON\>

34

35Rejectedblueprint:

36<REJECTED\_BLUEPRINT\_JSON\>

1Youareastrictdatasetreviewer\.Checkatravelhidden\-profile

2benchmarkforsemanticrealismandconsistency\.ReturnexactlyonevalidJSONobjectwithkeys

3accepted\(boolean\),issues\(arrayofconcisestrings\),andchecks\(object\)\.Donotrepairthetask\.

1Reviewthiscandidatebenchmarkafterdeterministicvalidation\.

2

3Rejectitifanypreferencestatementdisagreeswithitsstructuredpredicate;aplanattribute

4isvague,incomparable,orimplausible;aprivatefactleaksintopublictext;thedeclaredanswer

5ishintedbynames/order/wording;thescenarioneedsoutsideknowledge;constraintsarecontrived

6insteadofrealistic;anentiretravelerprofilehasnodecision\-relevantinformation;twooptions

7aresemanticallyindistinguishable;orthetaskcouldbesolvedfrompublicinformationorone

8profilealone\.Arealisticpreferencesharedbyallplansisallowedwhenthatsametravelerhas

9anotherpreferencethatdistinguishesplans\.

10

11Acceptonlyiftheexactfourplans,realisticprofiles,auniqueweighted\-utilityanswerunder

12must=\+2,prefer=\+1,avoid=\-1,reject=\-2,andmulti\-agentdependencyarecleartoacarefulhuman\.

13DonotrequireeverystrongpreferencetobesatisfiedinHiddenBenchmode;realisticcompromise

14plansareallowed\.

15

16Thedeterministicauditisauthoritativeforpredicateevaluation,scoring,winneruniqueness,

17andinformationdependency\.DoNOTrejectataskmerelybecauseanyoptionfailsaMUST,matchesa

18REJECT,orbecauseonetravelerlocallyranksthefinalanswerfirst\.Donotreinterpretoperators

19orreportoption\-levelscoreviolationsalreadyhandledbytheaudit\.Reviewonlysemantic

20fidelityofstatementstopredicates,realism,ambiguity,leakage,andunsupportedoutsidefacts\.

21

22Deterministicaudit:

23<DETERMINISTIC\_AUDIT\_JSON\>

24

25Blueprint:

26<CANDIDATE\_BLUEPRINT\_JSON\>