Mixture of Complementary Agents for Robust LLM Ensemble

# Mixture of Complementary Agents for Robust LLM Ensemble
Source: [https://arxiv.org/html/2605.24048](https://arxiv.org/html/2605.24048)
Kevin Lu Department of Mathematics, Rutgers University kll160@scarletmail\.rutgers\.eduYuang Zhang Department of Computer Science, George Mason University yzhang78@gmu\.eduJie Gao Lirong Xia Department of Computer Science, Rutgers University jg1555@cs\.rutgers\.edu, lirong\.xia@rutgers\.eduFang\-Yi Yu Department of Computer Science, George Mason University fangyiyu@gmu\.edu

###### Abstract

Multi\-AI collaboration—such as ensembling or debating large language models \(LLMs\)—is a promising paradigm for aggregating information and boosting performance\. A foundational step in these pipelines is to feed the responses of several*proposer*LLMs into a*summarizer*LLM, which synthesizes a better answer\. However, choosing which proposers to include is non\-trivial\. Existing approaches primarily focus either on accuracy \(picking the strongest models\) or diversity \(ensuring variety\), and often overlook the interactions among proposers and with the summarizer\. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its*complementarity*with others\. However, directly applying standard feature\-selection algorithms is impractical in the LLM setting due to prohibitive time complexity\. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy\-style selection algorithms that assess complementarity using a small labeled set\. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance–cost trade\-offs in practice\.

## 1Introduction

As today’s Large language model \(LLM\) ecosystem fragments into numerous models with diverse expertise, collaboration among LLMs has become promising and sometimes necessary for tackling emerging tasks such as mathematical reasoning\(Duet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib20)\), code generation\(Mahmudet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib37)\), and complex decision\-making\(Wuet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib38)\)\. A convenient instantiation is*ensemble after inference*, which aggregates the LLM outputs after the generation of full responses\. This includes well\-studied frameworks such as*LLM debating*\(Duet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib20); Estornell and Liu,[2024](https://arxiv.org/html/2605.24048#bib.bib23); Chanet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib25)\), in which multiple models iteratively exchange arguments before a final decision is reached, and*mixture\-of\-agents \(MoA\)*\(Wanget al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib18); Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8)\), which uses layered and summarization schemes to combine diverse model outputs\.

A fundamental step in the ensemble framework is inputting the responses ofNNLLM\-prompt pairs —the*proposers*111The same LLM under different prompts can be viewed as different proposers\.—into an aggregating LLM—the*summarizer*—which synthesizes a potentially improved answer\. Selecting which proposers to include is therefore critical: for a large proposer pool, it is impractical and inefficient to input responses from every available proposer due to context\-window limits and the degraded inference ability\(Liuet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib40)\)\. Existing methods often choose a small set of proposers based on their independent performance, primarily following two heuristics: \(i\)*accuracy\-seeking*—prioritize high\-accuracy proposers or even a single top model with multiple samples\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8); Jianget al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib15)\), and \(ii\)*diversity\-seeking*—explicitly mix heterogeneous outputs or prompts to avoid reinforcing similar mistakes\(Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7); Wanget al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib18)\)\.

However, both heuristics overlook a decisive factor: the complementarity of proposers both with one another and with the summarizer\. We argue these team effects, rather than individual quality or pairwise diversity, ultimately determine ensemble performance\. In particular, accuracy\-seeking methods rank proposers only by their individual performance, while diversity\-seeking methods reward variance regardless of quality\. We instead propose*mixture\-of\-complementary\-agents\(complementary\-MoA\)*—a framework that selects proposers for how well they work together as a team and with the summarizer\. The importance of complementarity can be observed from[Figure1](https://arxiv.org/html/2605.24048#S1.F1), which compares summarizer accuracy when inputting \(i\) the individually most accurate proposer versus \(ii\) the proposer that most complements the summarizer\. In this example, we consistently observe a nontrivial gap between the two choices, and furthermore, the most\-complementary proposer is sometimes weak on its own\. The upshot is both a promise and a challenge: as the ensemble sizekk\(the number of proposers selected for input\) grows, the optimal selection can yield substantial gains, yet it also complicates the search, since optimal teams cannot be inferred from individual performance alone\.

![Refer to caption](https://arxiv.org/html/2605.24048v1/figures/accuracy-complementarity-bar.png)Figure 1:Summarizer accuracies on AIME\(dolbokostya,[2025](https://arxiv.org/html/2605.24048#bib.bib36)\)when inputting themost accurateproposer vs\. themost complementaryproposer\. For each summarizerss, the proposer pool is \{Qwen3\-32B, Sky\-T1\-32B\-Preview, Aya\-expanse\-32B, Gemini\-1\.5\-Pro, Llama\-3\.3\-70B\-Instruct, AceReason\-Nemotron, and GPT\-4o\}, excludingssitself\.In this paper, we studyefficient proposer selection for multi\-LLM ensemblinggiven a summarizer, with a focus on selecting complementary proposers rather than merely strong or diverse ones\. Although proposer selection bears superficial resemblance to classical problems such as feature selection or data acquisition, the LLM setting introduces fundamentally new challenges: each evaluation of a candidate proposer team requires expensive summarizer calls, so wrapper\-style methods are computationally prohibitive at scale\. As a result, existing methods cannot be directly applied, and naive adaptations are prohibitively slow or brittle in practice\. See[Section2](https://arxiv.org/html/2605.24048#S2)for more discussions\.

Using multiple\-choice QA as a concrete testbed, we frame proposer selection as a complementarity\-driven optimization problem where the goal is to select a small ensemble that maximizes downstream accuracy while minimizing summarizer calls\. This formulation captures both cross\-model diversity and intra\-model prompt variation, which prior work has shown to be a major source of performance gains\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8); Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7)\)\.

Motivated by this perspective, we develop a family of proposer\-selection algorithms that navigate the accuracy–efficiency trade\-off\. We first introduce*model\-first greedy*, a wrapper\-style method that keeps the summarizer in the loop but reduces query complexity, measured by the total number of summarizer calls required for proposer selection\. We select the winning model at each step based on the average marginal gain its prompt variants provide to the current ensembleSS\. Once a model is selected, we identify its best\-performing prompt instance and add that model\-prompt pair toSS\. Model\-first greedy reduces query complexity based on the intuition that diversity across models matters more than diversity induced by different prompts within the same model\. To further reduce query complexity, we propose two algorithms that consider label\-level complementarity\.*Truth\-prediction greedy*selects proposers based on how well their reported labels help predict the ground truth;*oracle\-surrogate greedy*first fits a simple surrogate of the oracle and then selects proposers based on their marginal contributions measured by the surrogate model\. Both methods rely only on label\-level statistics and therefore require no—or only light—summarizer calls\.

We conduct an extensive empirical study across three popular reasoning benchmarks, spanning multiple proposer pools \(a dominating\-LLM regime and a mixed\-crowd regime\), different summarizers, and a range of ensemble sizes\. This evaluation reveals a consistent pattern: commonly used heuristics perform well only in narrow regimes and fail unpredictably elsewhere\. In contrast, our complementarity\-guided methods, including the truth\-prediction greedy which requires no summarizer call, are consistently robust across all scenarios\. Moreover, we frequently observe substantial gains from model\-first greedy over the strongest baseline, underscoring that explicitly optimizing for complementarity is crucial in ensemble frameworks\.

In summary, our main contributions are threefold:

- •We identify complementarity as a key, yet overlooked objective in agent\-level LLM ensembles and propose a more principled proposer\-selection framework, called complementary\-MoA, that explicitly optimizes it\.
- •Inspired by feature\-selection methods, we present three complementarity\-driven selection algorithms designed around LLM ensembles, where evaluating each proposer requires expensive model calls\. The resulting methods span a spectrum of accuracy–efficiency trade\-offs, giving practitioners a principled way to choose under different query budgets\.
- •Through large\-scale experiments, we provide a systematic and comprehensive comparison of LLM\-compatible proposer\-selection strategies, revealing the failure modes of existing heuristics and demonstrating that complementarity\-based selection delivers the most reliable performance across all tested settings\.

## 2Related Work

Agent\-Level Ensemble\.LLM ensembles can be constructed at multiple stages of the inference pipeline\(Chenet al\.,[2025b](https://arxiv.org/html/2605.24048#bib.bib13)\)\. We focus on*agent\-level*ensembling, which treats each LLM as a black box\. A closely related paradigm is*mixture\-of\-agents \(MoA\)*\(Wanget al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib18)\), a layered collaboration scheme in which, at a given layer, multiple proposers submit responses that are then aggregated by a summarizer\.Wanget al\.\([2024a](https://arxiv.org/html/2605.24048#bib.bib18)\)show that MoA effectively aggregates complementary signals, often yielding more reliable outputs than a single stronger model\. A follow\-up study challenges this design by demonstrating that repeatedly querying a single powerful LLM can also boost MoA\-style performance\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8)\)\. Another line of related work is*LLM debate*\(Duet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib20); Estornell and Liu,[2024](https://arxiv.org/html/2605.24048#bib.bib23); Chanet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib25); Wanget al\.,[2023a](https://arxiv.org/html/2605.24048#bib.bib24); Baeket al\.,[2026](https://arxiv.org/html/2605.24048#bib.bib55)\), where multiple models iteratively critique and refine one another’s responses that can often result in a consensus outperforming a single model\. However, asEstornell and Liu \([2024](https://arxiv.org/html/2605.24048#bib.bib23)\)point out, sharing all agents’ responses is not always optimal, where they observe that selecting a subset of LLMs that maximizes the mutual information between agents can be more effective\. Our work targets the foundational step in the above frameworks—theN→1N\\to 1summarization—with particular emphasis on proposing a more principled way to decide which proposers to select for the summarizer\.

Conditional mutual information is also a powerful tool for evaluating the informativeness of an agent’s marginal contribution beyond the current information\(Luet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib53); Zhanget al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib54)\)\. In concurrent work that applies a related idea to ensemble selection,Turkmenet al\.\([2026](https://arxiv.org/html/2605.24048#bib.bib52)\)explore complementarity in LLM ensembles by greedily maximizing the marginal mutual information between the proposer’s suggested labels and the ground truth\. While our high\-level objectives align, their method is restricted to binary labels and does not account for the synergy between proposers and the summarizer\. As illustrated in[Sections3\.2](https://arxiv.org/html/2605.24048#S3.SS2)and[5\.2](https://arxiv.org/html/2605.24048#S5.SS2), the summarizer significantly influences the composition of the optimal proposer set — a factor that label\-level selection algorithms cannot capture\.

Training\-Based Ensemble\.Prior literature has also explored the idea of training parametric meta\-models to decide, per query, which LLM \(or which LLM’s output\) to trust\. For example, fusion methods train a small network on features from multiple LLMs—e\.g\., concatenated probabilities or last\-layer embeddings—to predict the true label\(Jianget al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib15); Wanget al\.,[2023b](https://arxiv.org/html/2605.24048#bib.bib21)\)\. Routing methods learn a delegator that selects the most suitable agents for various tasks, e\.g\., RouteLLM uses human preference data to better trade off cost and quality\(Onget al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib28)\), and ZOOTER learns a router based on distilling rewards on training queriesLuet al\.\([2023](https://arxiv.org/html/2605.24048#bib.bib22)\)\. Similarly, cost\-aware cascades like FrugalGPT focus on learning when to use stronger but more expensive models\(Chenet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib29)\)\. Unlike prior training\-based ensembles, our framework avoids substantial supervised datasets: a few hundred examples suffice to learn the summarizer’s behavior for better proposer selection\. This light training also makes it compatible with closed\-source LLM summarizers, whereas past work either does not use an LLM summarizer or requires open\-source access \(e\.g\., logits/weights\)\.

Feature\-Selection\.Our problem is naturally relevant to feature selection, where the goal is to select a small subset of features that optimize the performance of an ML model\. One of the most classic example is Wrapper\(Kohavi and John,[1998](https://arxiv.org/html/2605.24048#bib.bib30)\), which evaluates features by repeatedly training a model—using forward/backward search\. The selection of features can also be implemented by inducing sparsity during training, with examples like LASSO\(Tibshirani,[1996](https://arxiv.org/html/2605.24048#bib.bib31)\)and LARS\(Kolter and Ng,[2009](https://arxiv.org/html/2605.24048#bib.bib32)\)\. Furthermore, it is often beneficial to filter likely weak features without retraining a predictor based on information\-theoretic \(e\.g\., mRMR\(Penget al\.,[2005](https://arxiv.org/html/2605.24048#bib.bib33)\)\) or neighborhood criteria \(e\.g\., Relief\(Urbanowiczet al\.,[2018](https://arxiv.org/html/2605.24048#bib.bib34)\)\)\. However, two challenges limit applicability to our setting\. First, wrapper\-style methods demand extensive retraining, while filter/embedded approaches operate only at the label level and thus ignore the LLM summarizer’s error\-correcting behavior\.222That said, in[Section5\.2](https://arxiv.org/html/2605.24048#S5.SS2)we show that even label\-level selection can produce more robust ensembles than existing baselines\.Second, the summarizer’s performance is often non\-monotone in the set of agents, making standard marginal\-gain scoring unreliable; this motivates new evaluation metrics for agent contribution—e\.g\., ourkk\-greedy algorithm in[Section4\.2](https://arxiv.org/html/2605.24048#S4.SS2)\.

## 3Problem Statement

We consider a dataset of multiple\-choice questions𝒬\\mathcal\{Q\}, and each questionq∈𝒬q\\in\\mathcal\{Q\}has a ground\-truth labelYq∈𝒴Y\_\{q\}\\in\\mathcal\{Y\}\. We assume true labels are available on a validation subset𝒬T⊂𝒬\\mathcal\{Q\}\_\{T\}\\subset\\mathcal\{Q\}of sizem=\|𝒬T\|m=\|\\mathcal\{Q\}\_\{T\}\|, while the remaining questions require inference \(test data\)\.

There areNN*proposers*\. Proposeriiprovides for questionqqa responseRi,q=\(Xi,q,Zi,q\)R\_\{i,q\}=\(X\_\{i,q\},Z\_\{i,q\}\), whereXi,q∈𝒴X\_\{i,q\}\\in\\mathcal\{Y\}is a proposed label andZi,qZ\_\{i,q\}is textual supporting reasoning \(e\.g\., chain\-of\-thought reasoning\)\. In our setting, we permit multiple proposers to originate from a single LLM by varying the prompt\. This is inspired by prior studies\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8); Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7)\), showing that feeding multiple responses from the same model to the summarizer can benefit the ensemble\. Letnpromptn\_\{\\mathrm\{prompt\}\}andnLLMn\_\{\\mathrm\{LLM\}\}be the number of prompts and models; the total number of proposers is thenN=nprompt⋅nLLMN=n\_\{\\mathrm\{prompt\}\}\\cdot n\_\{\\mathrm\{LLM\}\}\.

To improve accuracy, a*summarizer*aggregates multiple proposer responses, and outputs a potentially more accurate label\. Due to practical constraints \(e\.g\., LLMs often have strict input context limits\), we aim to select a \(small\) subset of proposers for the ensemble\. Formally, given the ensemble sizekkand a subsetS⊆\[N\]S\\subseteq\[N\]with\|S\|=k\|S\|=k, the summarizer outputsf​\(𝑹S,q\)f\(\\bm\{R\}\_\{S,q\}\), where𝑹S,q=\(Ri,q\)i∈S\\bm\{R\}\_\{S,q\}=\(R\_\{i,q\}\)\_\{i\\in S\}\. Both proposer and summarizer outputs are stochastic, and the key design choice is which proposers to select as input to the summarizer\.

We evaluate a selectionSSby the summarizer accuracy on test data:

Accf​\(S\)=1\|𝒬\\𝒬T\|​∑q∈𝒬\\𝒬TPr⁡\[f​\(𝑹S,q\)=Yq\]\.\\mathrm\{Acc\}\_\{f\}\(S\)=\\frac\{1\}\{\|\\mathcal\{Q\}\\backslash\\mathcal\{Q\}\_\{T\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\\backslash\\mathcal\{Q\}\_\{T\}\}\\Pr\\left\[f\(\\bm\{R\}\_\{S,q\}\)=Y\_\{q\}\\right\]\.The central problem is to choosekkout ofNNproposers to maximize accuracy, given summarizerff:

S∗=arg⁡maxS⊆\[N\],\|S\|=k⁡Accf​\(S\)\.S^\{\*\}=\\arg\\max\_\{S\\subseteq\[N\],\|S\|=k\}\\mathrm\{Acc\}\_\{f\}\(S\)\.\(1\)We study the trade\-offs introduced by the choice ofkk, though for clarity, most of our analysis proceeds while supposingkkis fixed\.

### 3\.1Previous Ideas

Label\-only aggregation\.The simplest approach aggregates only the discrete answers and ignores textual rationales\. A common choice is \(weighted\) majority voting over all proposers or a selected subset\. When proposers are conditionally independent with known accuracies, decision theory implies that a weighted majority rule \(with weights proportional to log\-odds of correctness\) is optimal\(Nitzan and Paroush,[1982](https://arxiv.org/html/2605.24048#bib.bib16)\)\. Our setting departs from these assumptions: LLM proposers exhibit strong dependencies, and their rationales carry additional signal\. Empirically, aggregation schemes that leverage an LLM summarizer to use rationales outperform simple majority vote on labels alone\(Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7); Tekinet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib17)\)\. Our experiments further confirm this point \(see[SectionB\.1](https://arxiv.org/html/2605.24048#A2.SS1)\)\.

Accuracy\-seeking aggregation\.A widely used heuristic in LLM ensembling is to select proposers by their estimated individual accuracy, where the intuition is that proposers with higher accuracy contribute more reliable evidence on new instances\. For example, one idea, called the*self\-MoA*, is to sample multiple diverse responses from the single best model and feed them to a summarizer\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8)\)\.

Diversity\-seeking aggregation\.A parallel line in the LLM ensemble literature argues that accuracy alone is insufficient: ensembles can benefit from diverse views\. This intuition inspired several suggestions that explicitly encourage diversity or strike an accuracy–diversity trade\-off \(e\.g\., maximize diversity conditioned on an accuracy bar\)\(Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7); Tekinet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib17); Wanget al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib18)\)\. However, for LLM ensembles, such diversity\-first strategies can be counterproductive: by admitting weak proposers in the name of variety, they often introduce low\-quality or correlated errors that depress the final aggregation performance\.

LLM\-as\-a\-Judge\.Another aggregation paradigm treats an LLM as a judge that evaluates, scores, or filters proposer responses and bases the final answer on the top\-rated candidates or their summary\. Such judge\-based approaches can provide effective supervision without ground truth and have shown strong empirical performance\(Liuet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib44)\)\. However, their effectiveness depends on the reliability and calibration of the judge model, and they may inherit systematic biases when judging responses from closely related LLMs\.

### 3\.2Limitations of Previous Ideas: A Motivating Counterexample

Below we provide a counterexample with four proposers and one Bayesian summarizer, whose signals areX1,X2,X3,X4X\_\{1\},X\_\{2\},X\_\{3\},X\_\{4\}andZfZ\_\{f\}respectively\. We show that atk=2k=2, accuracy\-seeking, mutual\-information\-seeking\(Turkmenet al\.,[2026](https://arxiv.org/html/2605.24048#bib.bib52)\), and diversity\-seeking algorithms all fail to identify the optimalS∗S^\{\*\}in[Equation1](https://arxiv.org/html/2605.24048#S3.E1)for ground truthYY\.

###### Proposition 1\.

There exists a joint distribution over\(Y,X1,X2,X3,X4,Zf\)\(Y,X\_\{1\},X\_\{2\},X\_\{3\},X\_\{4\},Z\_\{f\}\)and a Bayes\-optimal summarizerffsuch thatAccf​\(\{1,2\}\)=1\\mathrm\{Acc\}\_\{f\}\(\\\{1,2\\\}\)=1andAccf​\(S\)≤0\.9\\mathrm\{Acc\}\_\{f\}\(S\)\\leq 0\.9for every otherS⊆\{1,2,3,4\}S\\subseteq\\\{1,2,3,4\\\}with\|S\|=2\|S\|=2\. However, none of the following methods select\{1,2\}\\\{1,2\\\}:

1. 1\.Accuracy\-first:arg⁡max\|S\|=2​∑i∈SPr⁡\[Xi=Y\]\\arg\\max\_\{\|S\|=2\}\\sum\_\{i\\in S\}\\Pr\[X\_\{i\}=Y\]\.
2. 2\.Mutual\-information\-seeking:arg⁡max\|S\|=2⁡I​\(Y;XS\)\\arg\\max\_\{\|S\|=2\}I\(Y;X\_\{S\}\)\.
3. 3\.Diversity\-seeking:arg⁡maxi≠j⁡Pr⁡\[Xi≠Xj\]\\arg\\max\_\{i\\neq j\}\\Pr\[X\_\{i\}\\neq X\_\{j\}\]\.

The construction pits two types of proposer pairs against each other\. ProposersX3X\_\{3\}andX4X\_\{4\}are individually informative aboutYY\(each agrees withYYwith probability0\.80\.8\) and conditionally independent givenYY, making them look ideal under standard heuristics\. ProposersX1X\_\{1\}andX2X\_\{2\}, by contrast, are each marginally independent ofYYand highly correlated with one another, so they appear useless and redundant in isolation\. The catch is thatY=Zf⊕X1⊕X2Y=Z\_\{f\}\\oplus X\_\{1\}\\oplus X\_\{2\}, soX1X\_\{1\}andX2X\_\{2\}become exactly informative once combined with the summarizer’s private signalZfZ\_\{f\}\. Accuracy\-first and mutual\-information\-seeking methods both prefer\{3,4\}\\\{3,4\\\}becauseX1,X2X\_\{1\},X\_\{2\}have zero marginal accuracy and zero mutual information withYY; diversity\-seeking also prefers\{3,4\}\\\{3,4\\\}becauseX1,X2X\_\{1\},X\_\{2\}rarely disagree\. Yet\{1,2\}\\\{1,2\\\}is the unique size\-22set that lets the Bayes\-optimal summarizer recoverYYperfectly\.

Although this construction is admittedly artificial, the message it conveys is clear and general: any selection rule that evaluates proposers without reference toZfZ\_\{f\}\(the information of the summarizer\) can miss the complementarity that makes a selection optimal\. Our empirical results in[Figure1](https://arxiv.org/html/2605.24048#S1.F1)confirm this, motivating our framework of selecting proposers based on their complementarity with the summarizer beyond their individual quality or pairwise diversity\.

## 4Methods

Our central idea is to select proposers based on their collaborative performance with each other and with the summarizer—the selected proposers should complement their teammates\. In principle, one could exhaustively evaluate all size\-kkteams and pick the subset that maximizes summarizer accuracy\. In practice, searching over all\(Nk\)\\binom\{N\}\{k\}subsets is typically infeasible—e\.g\., even withN=20N=20andk=5k=5there are15,50415\{,\}504candidate teams—especially given the high inference cost of summarizing multi\-rationale inputs\.

An immediate idea is agreedy algorithm: we can iteratively find the proposer with the largest marginal contribution to the summarizer accuracy until we findkkproposers\. In particular, we initializeS0=∅S\_\{0\}=\\emptyset, and fort=1,…,kt=1,\\ldots,k, choose

it∈arg⁡maxi∈\[N\]\\St−1⁡\[Acc​\(St−1∪\{i\}\)−Acc​\(St−1\)\],i\_\{t\}\\in\\arg\\max\_\{i\\in\[N\]\\backslash S\_\{t\-1\}\}\\left\[\\mathrm\{Acc\}\(S\_\{t\-1\}\\cup\\\{i\\\}\)\-\\mathrm\{Acc\}\(S\_\{t\-1\}\)\\right\],then updateSt=St−1∪\{it\}S\_\{t\}=S\_\{t\-1\}\\cup\\\{i\_\{t\}\\\}\.

The performance of the greedy algorithm depends on the submodularity of the accuracy function, which in turn depends on the summarizer\. It turns out that for LLM summarizers, the accuracy function is not even monotone \(and thus not submodular\)—including a low\-accuracy proposer in the pool can actually reduce overall summarization performance\. This observation is supported by prior work\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8)\)and our experiments in[SectionB\.2](https://arxiv.org/html/2605.24048#A2.SS2)\. Therefore, in principle, the greedy algorithm can be far from the optimum in the worst case\. However, as we will see, the empirical performance of the \(simplified versions of\) the greedy algorithm is generally robust and significantly outperforms the baselines\.

Although the greedy algorithm is conceptually simple, it can be computationally demanding: it requires evaluating the accuracy functionO​\(k​N\)O\(kN\)times, which entailsO​\(k​N​m\)O\(kNm\)calls to the summarizer\. It is thus important to explore the trade\-off between ensemble accuracy and efficiency via some heuristic variants\. In our experiments, we implement only the simplified methods rather than the full greedy algorithm\.

### 4\.1Model\-First Greedy

Recall that a model and an instruction prompt determine a proposer\. However, responses generated by different prompts of the same model are typically more similar than responses generated by different models under the same prompt \(see[SectionB\.1](https://arxiv.org/html/2605.24048#A2.SS1)\)\. Inspired by hierarchical feature selection\(Ristoski and Paulheim,[2014](https://arxiv.org/html/2605.24048#bib.bib2)\), we introduce a simplification called*model\-first greedy*\. Unlike standard greedy—which estimates the marginal gain of every proposer using allmmquestions at each iteration —model\-first greedy scores allnpromptn\_\{\\mathrm\{prompt\}\}proposers from the same model using the common set ofmmquestions, then chooses a proposer within the best model\. Concretely, in iterationtt:

1. 1\.Partition𝒬T\\mathcal\{Q\}\_\{T\}into a training set𝒬Tt​r\\mathcal\{Q\}\_\{T\}^\{tr\}and a validation set𝒬Tv​a​l\\mathcal\{Q\}\_\{T\}^\{val\}for cross validation\.
2. 2\.For each modeli∈\[nLLM\]i\\in\[n\_\{\\mathrm\{LLM\}\}\], randomly assign each questionj∈𝒬Tt​rj\\in\\mathcal\{Q\}\_\{T\}^\{tr\}to one of the proposers associated with modelii, and estimate the accuracy of modeliiby averaging over the questions in the training set\.
3. 3\.Select the model with the highest estimated accuracy, and within that model, pick the proposer with the highest estimated accuracy from the previous step\.

Intuitively, the procedure prioritizes model selection while allowing more randomness in prompt selection\. This reduces summarizer calls per iteration fromN⋅mN\\cdot mtonLLM⋅mn\_\{\\mathrm\{LLM\}\}\\cdot m\.

### 4\.2Label\-level Complementarity

Model\-first greedy estimates each proposer’s marginal contribution via direct calls to the summarizer\. However, the proposers’ labels themselves carry predictive signals: the summarizer is more likely to answer correctly when it receives more correct inputs\. This motivates the idea of selecting proposers based on their*label\-level*information, which can improve scalability by avoiding extensive calls to the summarizer oracle\. This idea is related to*filter\-based*feature selection methods, e\.g\.\(Penget al\.,[2005](https://arxiv.org/html/2605.24048#bib.bib33); Urbanowiczet al\.,[2018](https://arxiv.org/html/2605.24048#bib.bib34)\), which remove likely weak features without retraining the predictor based on correlations between features\. In particular, we use an alternative set functionA​c​c^\\widehat\{Acc\}, defined with respect to a label\-based summarizergg, and use it to guide proposer selection\. This yields the following two methods\.

#### Truth\-Prediction Greedy

Built on the intuition that labels from a set of complementary proposers can predict the true label more accurately, we can train a light\-weight machine learning model to predictYqY\_\{q\}, and use it to select informative proposers\. Given a set of proposersSSand a family of models parametrized byθ∈Θ\\theta\\in\\Theta, we compute a valueAcc^gθ​\(S\)\\widehat\{\\mathrm\{Acc\}\}\_\{g\_\{\\theta\}\}\(S\)using the following procedure:

1. 1\.Partition𝒬T\\mathcal\{Q\}\_\{T\}into a training set𝒬Tt​r\\mathcal\{Q\}\_\{T\}^\{tr\}and a validation set𝒬Tv​a​l\\mathcal\{Q\}\_\{T\}^\{val\}for cross validation\.
2. 2\.Fitgθg\_\{\\theta\}\.Use the data in the training set,\(\(Xi,q\)i∈S,Yq\)q∈𝒬Tt​r\(\(X\_\{i,q\}\)\_\{i\\in S\},Y\_\{q\}\)\_\{q\\in\\mathcal\{Q\}\_\{T\}^\{tr\}\}to fit a modelgθg\_\{\\theta\}that maps\|S\|\|S\|labels on a question to a \(hard\) prediction of the ground truth label\. Here, proposers’ generated labels are viewed as features\.
3. 3\.Score proposer setSS\.On the validation set, evaluate the accuracy ofgθg\_\{\\theta\}using responses fromSSand returnAcc^gθ​\(S\)\\widehat\{\\mathrm\{Acc\}\}\_\{g\_\{\\theta\}\}\(S\)\.

Next, we select proposers using a variant of the greedy algorithm, called thekk\-greedy \(Alg\.[1](https://arxiv.org/html/2605.24048#algorithm1)\), usingAccgθ\\mathrm\{Acc\}\_\{g\_\{\\theta\}\}as the set function\. We first initialize a set of proposersS0=∅S\_\{0\}=\\emptyset\. Then, in roundt∈\{1,…,k\}t\\in\\\{1,\\ldots,k\\\}, unlike standard greedy—which estimates a candidate’s marginal gain relative to the current selected setSt−1S\_\{t\-1\}—kk\-greedy’s estimation always conditions on a set ofkkproposers\. The intuition is that LLM summarizers are non\-monotone, so an element that looks promising early can hurt performance at the final team sizekk\. Concretely, givenSt−1S\_\{t\-1\}, we randomly selectk−t\+1k\-t\+1proposers, to form a team of sizekk, denoted asLL\. Then, we measure candidateii’s contribution \(i∉St−1i\\notin S\_\{t\-1\}\) as the accuracy difference with and withoutii\(replacing one randomly chosen proposer inLL\)\. Averaging this difference over several random completions yields a more faithful estimate ofii’s value at the final team size\. We refer to this method as*truth\-prediction greedy*, which applies thekk\-greedy algorithm to the set functionAcc^gθ\\widehat\{\\mathrm\{Acc\}\}\_\{g\_\{\\theta\}\}\. We emphasize that truth\-prediction greedy relies on a lightweight ML model to guide proposer selection, but the final ensemble is still formed by feeding the chosen proposers into the LLM summarizer\.

Input:ground set

\[N\]\[N\], target size

kk, set function

Acc\\mathrm\{Acc\}, repetitions

MM
Output:selected set

SkS\_\{k\}
S0=∅S\_\{0\}=\\emptyset;

//initial selected proposers

for*t=1t=1tokk*do

for*i∈\[N\]∖St−1i\\in\[N\]\\setminus S\_\{t\-1\}*do

Set

Δi=0\\Delta\_\{i\}=0;

for*τ=1\\tau=1toMM*do

Sample

L⊆\[N\]∖\(St−1∪\{i\}\)L\\subseteq\[N\]\\setminus\(S\_\{t\-1\}\\cup\\\{i\\\}\)uniformly with

\|L\|=k−\|St−1\|\|L\|=k\-\|S\_\{t\-1\}\|;

//random completions

Sample

j∈Lj\\in Luniformly and set

L′←\(L∖\{j\}\)∪\{i\}L^\{\\prime\}\\leftarrow\(L\\setminus\\\{j\\\}\)\\cup\\\{i\\\};

Δi\+=Acc\(St−1∪L′\)−Acc\(St−1∪L\)\\Delta\_\{i\}\\mathrel\{\+\}=\\mathrm\{Acc\}\(S\_\{t\-1\}\\cup L^\{\\prime\}\)\-\\mathrm\{Acc\}\(S\_\{t\-1\}\\cup L\);

Δ^i​\(St−1\)=Δi/M\\widehat\{\\Delta\}\_\{i\}\(S\_\{t\-1\}\)=\\Delta\_\{i\}/M;

//estimated marginal

Choose

i⋆∈arg⁡maxi∈\[N\]∖St−1⁡Δ^i​\(St−1\)i^\{\\star\}\\in\\arg\\max\_\{i\\in\[N\]\\setminus S\_\{t\-1\}\}\\widehat\{\\Delta\}\_\{i\}\(S\_\{t\-1\}\);

St=St−1∪\{i⋆\}S\_\{t\}=S\_\{t\-1\}\\cup\\\{i^\{\\star\}\\\};

return

SkS\_\{k\};

Algorithm 1kk\-Greedy Proposer Selection w\.r\.t\.Acc\\mathrm\{Acc\}
#### Oracle\-Surrogate Greedy

Proposer selection under truth\-prediction greedy does not depend on the summarizer, so it may diverge from the ensemble’s true test performance\. As an alternative approach, we propose*oracle\-surrogate greedy*, where the idea is to fit a simple surrogate model to simulate the summarizer’s behavior using a small number of oracle queries on the training set, then use the surrogate to score and select proposers\. Although this method requires some summarizer calls for training, the surrogate model is kept simple as we only focus on label\-level information, making it more sample\-efficient than model\-first greedy in practice\.

We consider a surrogate modelg~\\tilde\{g\}based on the assumption that the summarizer’s accuracy depends primarily on how many of thekkinput labels are correct\. Specifically,g~:\{0,…,k\}→\[0,1\]\\tilde\{g\}:\\\{0,\\ldots,k\\\}\\rightarrow\[0,1\]maps a countccof correct labels to the expected summarizer accuracy when exactlyccout ofkkinputs are correct\. This implies that our surrogate model greatly reduces the query complexity by not distinguishing the proposer ID\. Given a set of proposersSS, the following procedure returns a valueAcc^g~​\(S\)\\widehat\{\\mathrm\{Acc\}\}\_\{\\tilde\{g\}\}\(S\)for setSS:

1. 1\.Partition𝒬T\\mathcal\{Q\}\_\{T\}into a training set𝒬Tt​r\\mathcal\{Q\}\_\{T\}^\{tr\}and a validation set𝒬Tv​a​l\\mathcal\{Q\}\_\{T\}^\{val\}for cross validation\.
2. 2\.Fitg~\\tilde\{g\}\.For eachc∈\{0,…,k\}c\\in\\\{0,\\ldots,k\\\}, repeatTg~T\_\{\\tilde\{g\}\}times: \(i\) sample a question from𝒬Ttr\\mathcal\{Q\}\_\{T\}^\{\\mathrm\{tr\}\}and a size\-kkset of proposers whose responses contain exactlycccorrect labels; \(ii\) query the summarizer on thesekkresponses\. Defineg~​\(c\)\\tilde\{g\}\(c\)as the empirical accuracy—i\.e\., the average correctness of the summarizer across theTg~T\_\{\\tilde\{g\}\}queries\.
3. 3\.Score proposer setSS\.For eachq∈𝒬Tv​a​lq\\in\\mathcal\{Q\}\_\{T\}^\{val\}, computecq​\(S\)c\_\{q\}\(S\), the number of correct labels inSS, and assignAcc^g~​\(S\)=1\|𝒬Tv​a​l\|​∑q∈𝒬Tv​a​lg~​\(cq​\(S\)\)\\widehat\{\\mathrm\{Acc\}\}\_\{\\tilde\{g\}\}\(S\)=\\frac\{1\}\{\|\\mathcal\{Q\}\_\{T\}^\{val\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\_\{T\}^\{val\}\}\\tilde\{g\}\(c\_\{q\}\(S\)\)\.

Next, we select a set ofkkagents by calling Alg\.[1](https://arxiv.org/html/2605.24048#algorithm1)withAcc^g~​\(S\)\\widehat\{\\mathrm\{Acc\}\}\_\{\\tilde\{g\}\}\(S\)as the set function\.

## 5Experiments

In this section, we first introduce the experimental setups, then we validate the proposed complementary\-MoA framework, diagnose the failure modes of baseline selectors, quantify efficiency–accuracy trade\-offs, and finally study prompting strategies for the summarizer that yield stronger ensembles\.

### 5\.1Experiment Setups

#### Dataset

We consider benchmark datasets with multi\-choice reasoning questions\. We use three popular reasoning datasets:AIME\(dolbokostya,[2025](https://arxiv.org/html/2605.24048#bib.bib36)\), CLadder\(Jinet al\.,[2023](https://arxiv.org/html/2605.24048#bib.bib35)\), and MMLU\-Pro\(Wanget al\.,[2024b](https://arxiv.org/html/2605.24048#bib.bib43)\)\.

AIMEconsists of about1,6001\{,\}600mathematical problems from competitions such as AIME and IMO\. The original answers are open\-ended integers; we convert each problem into a five\-choice question by randomly sampling four incorrect integers between0and1,0001\{,\}000\.CLaddercontains 10k causal reasoning questions that translate causal\-graph queries into natural\-language yes/no questions, spanning association, intervention, and counterfactual reasoning\.MMLU\-Prois a large\-scale multi\-choice benchmark targeting expert\-level reasoning across diverse academic domains, with harder questions and reduced answer leakage compared to earlier MMLU versions\.

#### Models

We consider a diverse set of LLMs: QwQ\-32B\(Team,[2025c](https://arxiv.org/html/2605.24048#bib.bib46)\), Qwen3\-32B\(Team,[2025b](https://arxiv.org/html/2605.24048#bib.bib47)\), Sky\-T1\-32B\-Preview\(Team,[2025a](https://arxiv.org/html/2605.24048#bib.bib48)\), aya\-expanse\-32B\(et al\.,[2024b](https://arxiv.org/html/2605.24048#bib.bib49)\), Gemini1\.5\-Pro\(et al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib50)\), Llama\-3\.3\-70B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib45)\), AceReason\-Nemotron\(Chenet al\.,[2025a](https://arxiv.org/html/2605.24048#bib.bib42)\), and GPT\-4o\(et al\.,[2024c](https://arxiv.org/html/2605.24048#bib.bib41)\)\. All models are run with a default temperature of 0\.7\. To reduce inference cost and latency, we disable chain\-of\-thought prompting\. For each of the LLMs, we considernprompt=5n\_\{\\mathrm\{prompt\}\}=5different prompts which are presented in[AppendixC](https://arxiv.org/html/2605.24048#A3), and each model\-prompt pair is viewed as a proposer\.

#### Settings

We evaluate ensemble performance across four factors—dataset, proposer pool, summarizer, and ensemble sizekk—using three datasets \(CLadder, AIME, MMLU\-Pro\), two pools \(with/without the strongest model\), multiple summarizers, and several choices ofk≤5k\\leq 5\. We use*complete pool*to refer to settings with all tested LLMs, and*reduced pool*to refer to settings where the best\-performing LLM is removed from the pool\. We use the latter to simulate a setting with mixed\-performing LLMs so as to improve the robustness of our results\. For example, “\(AIME, complete pool, Aya,k=3k=3\)” denotes the AIME dataset, a pool including the strongest LLM, Aya as summarizer, and selecting three proposers\. We limitkkto55because larger ensembles show diminishing returns\(Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7)\), while the cost of searching for the optimal team grows quickly\.

#### Baselines

Based on previous ideas in[Section3\.1](https://arxiv.org/html/2605.24048#S3.SS1), we consider the following baselines:

- •Input\-all: input allNNproposers\.333To fit the token limit, we truncate the responses from each proposer before summarizing\.
- •Best\-model: identify the most accurate model and select all proposers associated with it, in line with\(Liet al\.,[2025](https://arxiv.org/html/2605.24048#bib.bib8)\)\.
- •Top\-accuracy: select the most accuratekkproposers overall\.
- •MoA \(per\-model top\-11\): for each model, select the single most accurate proposer, inspired by the original mixture\-of\-agents framework\(Wanget al\.,[2024a](https://arxiv.org/html/2605.24048#bib.bib18)\)\.
- •Conditioned\-diversity: start with the most accurate proposer, then greedily add the proposer that maximizes average disagreement with the selected set, subject to an accuracy thresholdτ=0\.4\\tau=0\.4, inspired by\(Lauet al\.,[2024](https://arxiv.org/html/2605.24048#bib.bib7)\)\.
- •LLM\-judge: have a judge LLM \(Aya or GPT\-5\.2\) rate each proposer’s responses on a 1–5 scale, and select the top\-kkproposers by the average grade across the questions in training set\. Cross validation is applied in the same manner as our methods descried in[Section4](https://arxiv.org/html/2605.24048#S4)\.
- •Approximate Shapley: estimate each proposer’s Shapley value, which is its average marginal contribution to summarizer accuracy across coalitions of other proposers\. In particular, we sample4040random subsets of sizes11–44\(1010per size\) and2020random questions per subset, then averaging the accuracy gain from adding the proposer; select the top\-kkproposers by the estimated Shapley value\.

### 5\.2A Comparison of Proposer Selection Methods

We randomly selectm=400m=400questions for proposer selection and use the remaining questions for accuracy computing\. For each LLM, we iteratively usenprompt=5n\_\{\\mathrm\{prompt\}\}=5prompts to solicit responses for all the sampled questions, which returnsN=40N=40proposers’ responses for each question\. For each question, we randomize proposer order and include their individual accuracies in the instructions while inputting into the summarizer\. To further reduce the variance of the ensemble accuracy \(due to the randomness caused by the default temperature of LLMs\), we repeatedly call the summarizer ten times for each question in the test set and take the average\.

[Table1](https://arxiv.org/html/2605.24048#S5.T1)presents a summary of accuracy results across six settings withk=5k=5\. We tested two summarizers for each dataset while the better\-performed one is presented in the figure: Aya for AIME, and AceReason for Cladder and MMLUPro\. For the LLM\-judge baseline, we report results under two judges of contrasting strength: Aya, a relatively weaker judge, and GPT\-5\.2, a strong judge\. Detailed tables reporting the per\-method composition of selected proposers for each setting are deferred to[AppendixB](https://arxiv.org/html/2605.24048#A2)\.

Table 1:Accuracy comparison across all settings \(k=5k=5\)\. Per column, the best result is inbold blue, the second\-best is inblue, and the worst is inred\.AIME \(Aya\)Cladder \(AceReason\)MMLU\-Pro \(AceReason\)MethodCompleteReducedCompleteReducedCompleteReducedInput\-all0\.6580\.6210\.8010\.7900\.7240\.687Best\-model0\.3490\.3320\.7860\.7930\.7220\.664Top\-accuracy0\.3770\.3640\.7770\.7800\.7460\.666MoA0\.5800\.5620\.7570\.7600\.7370\.687Conditioned\-diversity0\.4830\.4680\.7420\.7650\.6830\.669Aya\-judge0\.4350\.4150\.7200\.7480\.7100\.692GPT5\.2\-judge0\.4480\.4500\.7600\.7630\.5650\.568Truth\-prediction Greedy0\.6110\.5220\.7620\.7610\.7550\.678Oracle\-surrogate Greedy0\.6070\.4970\.7520\.7650\.7650\.687Model\-first Greedy0\.6540\.6320\.8120\.8020\.7380\.711#### Importance of Complementarity

First, our results show that accuracy\-seeking and diversity\-seeking baselines each fail in complementary ways\. Accuracy\-seeking methods \(Top\-accuracy, Best\-model\) perform poorly on AIME even when the pool contains a clearly strong proposer \(Gemini1\.5\-pro\)\. Consistent with[Figure1](https://arxiv.org/html/2605.24048#S1.F1), this indicates that the most accurate proposer is not necessarily the most complementary to the summarizer\. Diversity\-seeking methods \(Conditioned\-diversity\) instead fail on MMLU\-Pro, where Sky alone composes well with the summarizer and adding diverse but lower\-accuracy proposers degrades aggregation\. In both cases, the failure stems from optimizing a proxy—individual accuracy or pairwise diversity—that does not capture how proposers interact with the summarizer\.

In contrast, the label\-level complementarity\-aware methods \(Truth\-prediction Greedy and Oracle\-surrogate Greedy\), while not always the top performers, deliver consistently strong performance across all settings\. Model\-first Greedy goes further: it ranks among the top two methods in five of the six settings in[Table1](https://arxiv.org/html/2605.24048#S5.T1)\. Together, these findings indicate that explicitly accounting for complementarity is crucial for effective LLM ensembles\.

The LLM\-judge baseline performs poorly across all datasets\. This is consistent with our main message: the judge evaluates each proposer in isolation rather than as a team interacting with the summarizer, and its grades can additionally inherit biases from the judge model itself\. Notably, using a stronger judge does not necessarily yield a stronger ensemble—GPT5\.2\-judge is significantly outperformed by the weaker Aya\-judge—reinforcing that individual\-quality scoring is an unreliable proxy for ensemble value\.

The approximate Shapley baseline is substantially more expensive than the other baselines, as it requires repeatedly sampling proposer subsets and isolating each proposer’s marginal contribution\. We therefore evaluate it only in the \(AIME,⋅\\cdot, Aya,k=5k=5\) setting for reference\. As shown in[Tables12](https://arxiv.org/html/2605.24048#A2.T12)and[13](https://arxiv.org/html/2605.24048#A2.T13), its accuracy is consistently dominated by Model\-first Greedy, indicating that Model\-first Greedy offers a better trade\-off between accuracy and query complexity\.

*What explains this performance difference?*[Figure2](https://arxiv.org/html/2605.24048#S5.F2)presents the empirical distributions of the number of correct labelsc∈\{0,…,k\}c\\in\\\{0,\\ldots,k\\\}obtained by the selected proposers under three representative methods on the AIME dataset with reduced pool\. The bar corresponding toccindicates the fraction of questions in the dataset where exactccselected proposers have answered correctly\. The overlaid curve shows the summarizer accuracy conditioned onccout ofkkinput responses being correct\.444To reduce variance, we pool samples from all proposers to estimate the conditioned accuracy\. Hence, the curve is identical across methods within the same setting\.

Clear patterns emerge: accuracy\-seeking baselines, such as Best\-model, induce a U\-shaped distribution ofcc, while diversity\-seeking baselines, such as Conditioned\-diversity, exhibit a bell\-shaped distribution\. This indicates that Best\-model tends to select proposers who make similar mistakes, which can be problematic when the summarizer accuracy curve is concave—i\.e\., when the marginal benefit of additional correct answers diminishes\. However, Conditioned\-diversity concentrates mass aroundc=⌊k/2⌋c=\\lfloor k/2\\rfloorby seeking different proposers, which can be suboptimal when the summarizer requires a strong majority to achieve a significant accuracy boost\. In contrast, complementarity\-based methods yield distributions that lie between these two extremes, illustrating their robustness across different summarizer behaviors\. This observation echoes our theoretical insights present in[Section3\.2](https://arxiv.org/html/2605.24048#S3.SS2)\.

![Refer to caption](https://arxiv.org/html/2605.24048v1/x1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.24048v1/x2.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.24048v1/x3.png)\(c\)

Figure 2:Distribution of the number of correct answers \(bars\) and summarizer accuracyAccf​\(c\)\\mathrm\{Acc\}\_\{f\}\(c\)\(line\) for three exemplary methods in the \(AIME, reduced pool, Aya,k=5k=5\) setting\.
#### Efficiency\-Accuracy Tradeoff

Table 2:Query complexity of considered methods, measured as the number of summarizer queries during proposer\-selection\.MethodComplexityApproximate Shapley2​nLLM​nprompt​z​Th2\\,n\_\{\\mathrm\{LLM\}\}\\,n\_\{\\mathrm\{prompt\}\}\\,z\\,T\_\{h\}Truth\-Prediction Greedy0Oracle\-Surrogate Greedy\(k\+1\)​Tg~\(k\+1\)\\,T\_\{\\tilde\{g\}\}Model\-First GreedynLLM​m​kn\_\{\\mathrm\{LLM\}\}\\,m\\,kOther methods0
SymbolMeaningnLLMn\_\{\\mathrm\{LLM\}\}number of candidate LLMsnpromptn\_\{\\mathrm\{prompt\}\}number of prompts per LLMkknumber of selected proposersmmsize of training setz,Th,Tg~z,T\_\{h\},T\_\{\\tilde\{g\}\}method\-specific Monte Carlo sample size

We quantify each method’s efficiency by thenumber of summarizer callsmade during proposer selection \(i\.e\., the query complexity\)\. Calls used by individual proposers to generate responses are excluded, as they are identical across methods and do not affect relative efficiency\. We summarize each method’s complexity below, with formulas shown in Table[2](https://arxiv.org/html/2605.24048#S5.T2)\.

- •LLM\-judge baselines\.These methods issue no summarizer calls but requiremmcalls to a judge LLM to score proposer responses on the training set\. We treat judge calls as substantially cheaper, since each produces only short numerical ratings, whereas a summarizer call must reason over a long, multi\-rationale context and emit a full answer\.
- •Oracle\-Surrogate Greedy\.Fits the summarizer’s accuracy by drawingTg~T\_\{\\tilde\{g\}\}Monte Carlo samples for each of the\(k\+1\)\(k\+1\)possible correct\-count cases \(from0tokk\), resulting in\(k\+1\)​Tg~\(k\+1\)\\,T\_\{\\tilde\{g\}\}calls; withTg~=200T\_\{\\tilde\{g\}\}=200andk=5k=5, this is1,2001\{,\}200calls\.
- •Model\-First Greedy\.Evaluates each of thenLLMn\_\{\\mathrm\{LLM\}\}models onmmtraining questions in each ofkkrounds, givingnLLM⋅m⋅kn\_\{\\mathrm\{LLM\}\}\\cdot m\\cdot kcalls\. In our experiments, this is8⋅400⋅5=16,0008\\cdot 400\\cdot 5=16\{,\}000calls\.
- •Approximate Shapley\.Estimates each proposer’s Shapley value by random coalition sampling\. For each of theN=nLLM⋅npromptN=n\_\{\\mathrm\{LLM\}\}\\cdot n\_\{\\mathrm\{prompt\}\}proposersii, we samplez=10​\(k−1\)z=10\(k\-1\)coalitions of other proposers \(10 per coalition size, for sizes11tok−1k\-1\), and for each sampled coalitionSSwe query the summarizer onTh=20T\_\{h\}=20training questions twice—once onSSand once onS∪\{i\}S\\cup\\\{i\\\}\. The total is2⋅N⋅z⋅Th2\\cdot N\\cdot z\\cdot T\_\{h\}calls; withk=5k=5, this is2⋅40⋅40⋅20=64,0002\\cdot 40\\cdot 40\\cdot 20=64\{,\}000calls\.
- •Other methods\.Other baselines and Truth\-Prediction Greedy rely on proposers’ individual reported labels and thus incur zero LLM calls during proposer selection\.

### 5\.3Prompting Summarizer

Beyond which proposers are selected, how their responses are presented to the summarizer also shapes aggregation quality\. We examine two prompting choices: \(i\) the ordering of proposer responses and \(ii\) whether each proposer’s individual accuracy is disclosed to the summarizer\.

We pick five proposers with relatively large accuracy differences, and input their responses to the summarizer in ascending, descending, or randomized order of individual accuracy\. For each case, we further distinguish two settings depending on whether the accuracy of each proposer is input to the summarizer as a part of the prompt\.

Table[3](https://arxiv.org/html/2605.24048#S5.T3)presents an example with two key takeaways\. First,inputting accuracy matters\.Providing per\-proposer accuracies affects performance in opposite ways across datasets—improving on AIME and MMLU\-Pro yet degrading on CLadder\. This suggests that the LLM summarizer can respond to the “reliability” information, but the net effect is heavily context\-dependent\. When accuracy information helps, the gain is largest under random ordering — consistent with the summarizer relying more on explicit reliability cues when no positional signal is available\.

Second,ordering matters\.Placing stronger proposers later in the prompt \(using ascending accuracy order\) outperforms descending order\. This pattern is consistent with recency bias in long\-context LLM inference\(Peysakhovich and Lerer,[2023](https://arxiv.org/html/2605.24048#bib.bib51)\): earlier content tends to receive less attention relative to later content\. These findings help clarify why our main experiments adopted a randomized ordering with per\-proposer accuracies\.

Table 3:Summarizer accuracies under different proposer orderings and whether individual accuracies are input in the \(⋅\\cdot,⋅\\cdot, AceReason,k=5k=5\) setting with proposers: QwQ, Gemini, Llama, GPT, Aya, under instruction prompt 1 \([AppendixC](https://arxiv.org/html/2605.24048#A3)\)\.AIMECLadderMMLU\-ProOrderingno accwith accno accwith accno accwith accAscending0\.5240\.5260\.8060\.7730\.4060\.449Descending0\.5000\.4960\.7920\.7590\.3850\.449Randomized0\.4980\.5380\.7980\.7740\.4010\.448

## 6Conclusion and Discussion

We proposecomplementary\-MoA, a proposer\-selection framework for post\-inference LLM ensembles built around an observation that prior work has largely missed: an optimal ensemble depends not only on proposers’ individual accuracy or mutual diversity, but on how well they complement both each other and the summarizer\. Existing heuristics—whether based on individual accuracy or pairwise diversity—evaluate proposers in isolation from the summarizer and therefore overlook this structural factor in aggregation\. We make the gap precise both theoretically and empirically:[Proposition1](https://arxiv.org/html/2605.24048#Thmproposition1)constructs an example in which accuracy\-, mutual\-information\-, and diversity\-seeking selectors all simultaneously fail to recover the optimal proposer set, under a Bayes\-optimal summarizer, and our experiments confirm that this is not a contrived artifact but a recurring failure mode in practice\.

Building on this insight, we connect proposer selection to feature selection over a black\-box objective and instantiate three algorithms—model\-first, truth\-prediction, and oracle\-surrogate greedy—that span a spectrum of accuracy–efficiency trade\-offs\. Across three reasoning benchmarks, multiple summarizers, and varying proposer pools, our methods are consistently robust, clarify when and why standard baselines fail, and motivate prompting strategies that further strengthen multi\-LLM collaboration\.

We acknowledge several limitations and directions for future work\. First, our label\-level algorithms target multiple\-choice tasks; extending them to open\-ended generation requires a reliable evaluation metric that captures partial correctness\. Second, the efficiency–accuracy frontier is not yet fully charted, where hybrid designs \(e\.g\., selecting the firstk′<kk^\{\\prime\}<kproposers via complementarity and filling the remainder by accuracy\) may further reduce query complexity\.

## Acknowledgements

Lu and Gao would like to acknowledge NSF support through IIS\-2229876, DMS\-2220271, CNS\-2515159, DMS\-2311064, and CCF\-2118953\. Xia acknowledges NSF \#2450124, \#2517733, and \#2518373 for support\.

## References

- D\. Baek, A\. Estornell, Y\. Zhang, M\. F\. Taufiq, J\. Ton, J\. Mei, and T\. Wang \(2026\)D\-FUSEr: diverse failure, unified success via error\-distribution shaping in LLM reasoning\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu \(2023\)Chateval: towards better LLM\-based evaluators through multi\-agent debate\.arXiv preprint arXiv:2308\.07201\.Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1),[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- L\. Chen, M\. Zaharia, and J\. Zou \(2023\)FrugalGPT: how to use large language models while reducing cost and improving performance\.arXiv preprint arXiv:2305\.05176\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p3.1)\.
- Y\. Chen, Z\. Yang, Z\. Liu, C\. Lee, P\. Xu, M\. Shoeybi, B\. Catanzaro, and W\. Ping \(2025a\)AceReason\-nemotron: advancing math and code reasoning through reinforcement learning\.arXiv preprint arXiv:2505\.16400\.External Links:[Link](https://arxiv.org/abs/2505.16400)Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. Chen, J\. Li, P\. Chen, Z\. Li, K\. Sun, Y\. Luo, Q\. Mao, D\. Yang, H\. Sun, and P\. S\. Yu \(2025b\)Harnessing multiple large language models: a survey on LLM ensemble\.External Links:2502\.18036Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- dolbokostya \(2025\)Math problems with answers \(AIME, IMO\)\.Note:[https://www\.kaggle\.com/datasets/dolbokostya/math\-problems\-with\-answers\-aime\-imo](https://www.kaggle.com/datasets/dolbokostya/math-problems-with-answers-aime-imo)Kaggle dataset; accessed 2025\-09\-23Cited by:[Figure 1](https://arxiv.org/html/2605.24048#S1.F1),[Figure 1](https://arxiv.org/html/2605.24048#S1.F1.4.2),[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2023\)Improving factuality and reasoning in language models through multiagent debate\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1),[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- A\. Estornell and Y\. Liu \(2024\)Multi\-LLM debate: framework, principals, and interventions\.Advances in Neural Information Processing Systems37,pp\. 28938–28964\.Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1),[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- G\. T\. et al\. \(2024a\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530\.External Links:[Link](https://arxiv.org/abs/2403.05530)Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- J\. D\. et al\. \(2024b\)Aya expanse: combining research breakthroughs for a new multilingual frontier\.arXiv preprint arXiv:2412\.04261\.External Links:[Link](https://arxiv.org/abs/2412.04261)Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- O\. et al\. \(2024c\)GPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Jiang, X\. Ren, and B\. Y\. Lin \(2023\)LLM\-Blender: ensembling large language models with pairwise ranking and generative fusion\.External Links:2306\.02561Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p2.1),[§2](https://arxiv.org/html/2605.24048#S2.p3.1)\.
- Z\. Jin, Y\. Chen, F\. Leeb, L\. Gresele, O\. Kamal, Z\. Lyu, K\. Blin, F\. G\. Adauto, M\. Kleiman\-Weiner, M\. Sachan, and B\. Schölkopf \(2023\)CLadder: assessing causal reasoning in language models\.External Links:2312\.04350Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px1.p1.1)\.
- R\. Kohavi and G\. H\. John \(1998\)The wrapper approach\.InFeature Extraction, Construction and Selection: a data mining perspective,pp\. 33–50\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p4.1)\.
- J\. Z\. Kolter and A\. Y\. Ng \(2009\)Regularization and feature selection in least\-squares temporal difference learning\.InProceedings of the 26th Annual International Conference on Machine Learning,ICML ’09,New York, NY, USA,pp\. 521–528\.External Links:ISBN 9781605585161,[Link](https://doi.org/10.1145/1553374.1553442),[Document](https://dx.doi.org/10.1145/1553374.1553442)Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p4.1)\.
- G\. K\. R\. Lau, W\. Hu, D\. Liu, J\. Chen, S\. Ng, and B\. K\. H\. Low \(2024\)Dipper: diversity in prompts for producing large language model ensembles in reasoning tasks\.External Links:2412\.15238,[Link](https://arxiv.org/abs/2412.15238)Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p2.1),[§1](https://arxiv.org/html/2605.24048#S1.p5.1),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p3.1),[§3](https://arxiv.org/html/2605.24048#S3.p2.9),[5th item](https://arxiv.org/html/2605.24048#S5.I1.i5.p1.1),[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px3.p1.5)\.
- W\. Li, Y\. Lin, M\. Xia, and C\. Jin \(2025\)Rethinking mixture\-of\-agents: is mixing different large language models beneficial?\.External Links:2502\.00674,[Link](https://arxiv.org/abs/2502.00674)Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1),[§1](https://arxiv.org/html/2605.24048#S1.p2.1),[§1](https://arxiv.org/html/2605.24048#S1.p5.1),[§2](https://arxiv.org/html/2605.24048#S2.p1.1),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p2.1),[§3](https://arxiv.org/html/2605.24048#S3.p2.9),[§4](https://arxiv.org/html/2605.24048#S4.p3.1),[2nd item](https://arxiv.org/html/2605.24048#S5.I1.i2.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2023\)Lost in the middle: how language models use long contexts\.External Links:2307\.03172Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p2.1)\.
- Z\. Liu, Y\. Zhang, P\. Li, Y\. Liu, and D\. Yang \(2024\)A dynamic llm\-powered agent network for task\-oriented agent collaboration\.External Links:2310\.02170,[Link](https://arxiv.org/abs/2310.02170)Cited by:[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p4.1)\.
- K\. Lu, H\. Yuan, R\. Lin, J\. Lin, Z\. Yuan, C\. Zhou, and J\. Zhou \(2023\)Routing to the expert: efficient reward\-guided ensemble of large language models\.External Links:2311\.08692,[Link](https://arxiv.org/abs/2311.08692)Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p3.1)\.
- Y\. Lu, S\. Xu, Y\. Zhang, Y\. Kong, and G\. Schoenebeck \(2024\)Eliciting informative text evaluations with large language models\.InProceedings of the 25th ACM Conference on Economics and Computation,EC ’24,New York, NY, USA,pp\. 582–612\.External Links:ISBN 9798400707049,[Link](https://doi.org/10.1145/3670865.3673532),[Document](https://dx.doi.org/10.1145/3670865.3673532)Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p2.1)\.
- T\. Mahmud, B\. Duan, C\. Pasareanu, and G\. Yang \(2025\)Enhancing LLM code generation with ensembles: a similarity\-based selection approach\.arXiv preprint arXiv:2503\.15838\.Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1)\.
- S\. Nitzan and J\. Paroush \(1982\)Optimal decision rules in uncertain dichotomous choice situations\.International Economic Review,pp\. 289–297\.Cited by:[§B\.1](https://arxiv.org/html/2605.24048#A2.SS1.p2.2),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p1.1)\.
- I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica \(2024\)Routellm: learning to route llms with preference data\.arXiv preprint arXiv:2406\.18665\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p3.1)\.
- H\. Peng, F\. Long, and C\. Ding \(2005\)Feature selection based on mutual information criteria of max\-dependency, max\-relevance, and min\-redundancy\.IEEE Transactions on pattern analysis and machine intelligence27\(8\),pp\. 1226–1238\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p4.1),[§4\.2](https://arxiv.org/html/2605.24048#S4.SS2.p1.2)\.
- A\. Peysakhovich and A\. Lerer \(2023\)Attention sorting combats recency bias in long context language models\.External Links:2310\.01427Cited by:[§5\.3](https://arxiv.org/html/2605.24048#S5.SS3.p4.1)\.
- P\. Ristoski and H\. Paulheim \(2014\)Feature selection in hierarchical feature spaces\.InInternational conference on discovery science,pp\. 288–300\.Cited by:[§4\.1](https://arxiv.org/html/2605.24048#S4.SS1.p1.4)\.
- N\. Team \(2025a\)Sky\-T1: fully open\-source reasoning model with o1\-preview performance in $450 budget\.Note:[https://novasky\-ai\.github\.io/posts/sky\-t1](https://novasky-ai.github.io/posts/sky-t1)Accessed: 2025\-01\-09Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- Q\. Team \(2025b\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- Q\. Team \(2025c\)Qwq\-32b: embracing the power of reinforcement learning\.March\.Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px2.p1.1)\.
- S\. F\. Tekin, F\. Ilhan, T\. Huang, S\. Hu, and L\. Liu \(2024\)LLM\-TOPLA: efficient LLM ensemble by maximising diversity\.arXiv preprint arXiv:2410\.03953\.Cited by:[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p3.1)\.
- R\. Tibshirani \(1996\)Regression shrinkage and selection via the LASSO\.Journal of the Royal Statistical Society Series B: Statistical Methodology58\(1\),pp\. 267–288\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p4.1)\.
- Y\. Turkmen, B\. Buyukates, and M\. Bastopcu \(2026\)Don’t always pick the highest\-performing model: an information theoretic view of llm ensemble selection\.External Links:2602\.08003,[Link](https://arxiv.org/abs/2602.08003)Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p2.1),[§3\.2](https://arxiv.org/html/2605.24048#S3.SS2.p1.5)\.
- R\. J\. Urbanowicz, M\. Meeker, W\. La Cava, R\. S\. Olson, and J\. H\. Moore \(2018\)Relief\-based feature selection: introduction and review\.Journal of biomedical informatics85,pp\. 189–203\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p4.1),[§4\.2](https://arxiv.org/html/2605.24048#S4.SS2.p1.2)\.
- B\. Wang, X\. Yue, and H\. Sun \(2023a\)Can ChatGPT defend its belief in truth? evaluating LLM reasoning via debate\.External Links:2305\.13160,[Link](https://arxiv.org/abs/2305.13160)Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p1.1)\.
- H\. Wang, F\. M\. Polo, Y\. Sun, S\. Kundu, E\. Xing, and M\. Yurochkin \(2023b\)Fusing models with complementary expertise\.arXiv preprint arXiv:2310\.01542\.Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p3.1)\.
- J\. Wang, J\. Wang, B\. Athiwaratkun, C\. Zhang, and J\. Zou \(2024a\)Mixture\-of\-agents enhances large language model capabilities\.arXiv preprint arXiv:2406\.04692\.Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1),[§1](https://arxiv.org/html/2605.24048#S1.p2.1),[§2](https://arxiv.org/html/2605.24048#S2.p1.1),[§3\.1](https://arxiv.org/html/2605.24048#S3.SS1.p3.1),[4th item](https://arxiv.org/html/2605.24048#S5.I1.i4.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024b\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[§5\.1](https://arxiv.org/html/2605.24048#S5.SS1.SSS0.Px1.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2023\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.External Links:2308\.08155Cited by:[§1](https://arxiv.org/html/2605.24048#S1.p1.1)\.
- Y\. Zhang, J\. Pang, Z\. Zhu, and Y\. Liu \(2025\)Evaluating LLM\-contaminated crowdsourcing data without ground truth\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2605.24048#S2.p2.1)\.

## Appendix AProofs

###### Proof of[Proposition1](https://arxiv.org/html/2605.24048#Thmproposition1)\.

We consider binary signals in\{0,1\}\\\{0,1\\\}\. LetX1X\_\{1\}andZfZ\_\{f\}be independent and uniform on\{0,1\}\\\{0,1\\\}, and setX2X\_\{2\}so thatPr⁡\[X2=X1\]=0\.9\\Pr\[X\_\{2\}=X\_\{1\}\]=0\.9\. Define the ground truth asY=Zf⊕X1⊕X2Y=Z\_\{f\}\\oplus X\_\{1\}\\oplus X\_\{2\}\. LetX3,X4X\_\{3\},X\_\{4\}be conditionally independent givenYYwithPr⁡\[X3=Y\]=Pr⁡\[X4=Y\]=0\.8\\Pr\[X\_\{3\}=Y\]=\\Pr\[X\_\{4\}=Y\]=0\.8\. The summarizer privately observesZfZ\_\{f\}, and proposeriireportsXiX\_\{i\}fori=1,…,4i=1,\\dots,4\.

#### Optimality of\{1,2\}\\\{1,2\\\}\.

BecauseYYis a deterministic function of\(Zf,X1,X2\)\(Z\_\{f\},X\_\{1\},X\_\{2\}\), the Bayes\-optimal summarizer recoversYYexactly from\(Zf,X1,X2\)\(Z\_\{f\},X\_\{1\},X\_\{2\}\), soAccf​\(\{1,2\}\)=1\\mathrm\{Acc\}\_\{f\}\(\\\{1,2\\\}\)=1\. Every other size\-22set contains at most one of\{1,2\}\\\{1,2\\\}and hence cannot recoverYYexactly\.

ForS=\{3,4\}S=\\\{3,4\\\}, sinceZfZ\_\{f\}alone is independent ofYY, it provides no information withoutX1X\_\{1\}orX2X\_\{2\}\. The Bayes\-optimalfftherefore aggregatesX3,X4X\_\{3\},X\_\{4\}alone, agreeing on the correct value with probability0\.640\.64and tie\-breaking on disagreement, for total accuracy

0\.64⋅1\+0\.32⋅12=0\.8\.0\.64\\cdot 1\+0\.32\\cdot\\tfrac\{1\}\{2\}=0\.8\.
For\|S∩\{1,2\}\|=\|S∩\{3,4\}\|=1\|S\\cap\\\{1,2\\\}\|=\|S\\cap\\\{3,4\\\}\|=1, without loss of generality, takeS=\{1,3\}S=\\\{1,3\\\}\. By symmetry it suffices to compute the conditional accuracy givenZf=X1=0Z\_\{f\}=X\_\{1\}=0, and the other three combinations are identical\. Under this conditioning,Y=X2Y=X\_\{2\}, soPr⁡\[Y=0∣Zf=0,X1=0\]=0\.9\\Pr\[Y=0\\mid Z\_\{f\}=0,X\_\{1\}=0\]=0\.9\. The summarizer also observesX3X\_\{3\}withPr⁡\[X3=Y∣Y\]=0\.8\\Pr\[X\_\{3\}=Y\\mid Y\]=0\.8\. By Bayes’ rule,

Pr⁡\[X3=0∣Zf=0,X1=0\]=0\.9⋅0\.8\+0\.1⋅0\.2=0\.74,\\Pr\[X\_\{3\}=0\\mid Z\_\{f\}=0,X\_\{1\}=0\]=0\.9\\cdot 0\.8\+0\.1\\cdot 0\.2=0\.74,and the posteriors onYYare

Pr⁡\[Y=0∣Zf=0,X1=0,X3=0\]\\displaystyle\\Pr\[Y=0\\mid Z\_\{f\}=0,X\_\{1\}=0,X\_\{3\}=0\]=0\.9⋅0\.80\.74=3637,\\displaystyle=\\frac\{0\.9\\cdot 0\.8\}\{0\.74\}=\\frac\{36\}\{37\},Pr⁡\[Y=0∣Zf=0,X1=0,X3=1\]\\displaystyle\\Pr\[Y=0\\mid Z\_\{f\}=0,X\_\{1\}=0,X\_\{3\}=1\]=0\.9⋅0\.20\.26=913\.\\displaystyle=\\frac\{0\.9\\cdot 0\.2\}\{0\.26\}=\\frac\{9\}\{13\}\.The Bayes\-optimal outputs0regardless ofX3X\_\{3\}, and the conditional accuracy given\(Zf=0,X1=0\)\(Z\_\{f\}=0,X\_\{1\}=0\)isPr⁡\[Y=0∣Zf=0,X1=0\]=0\.9\\Pr\[Y=0\\mid Z\_\{f\}=0,X\_\{1\}=0\]=0\.9\. HenceAccf​\(\{1,3\}\)=0\.9\\mathrm\{Acc\}\_\{f\}\(\\\{1,3\\\}\)=0\.9, and by symmetry the same holds for\{1,4\},\{2,3\},\{2,4\}\\\{1,4\\\},\\\{2,3\\\},\\\{2,4\\\}\.

#### Failure of the three methods\.

SinceYYis uniform conditional on\(X1,X2\)\(X\_\{1\},X\_\{2\}\), we havePr⁡\[X1=Y\]=Pr⁡\[X2=Y\]=0\.5\\Pr\[X\_\{1\}=Y\]=\\Pr\[X\_\{2\}=Y\]=0\.5andI​\(Y;X1,X2\)=0I\(Y;X\_\{1\},X\_\{2\}\)=0\. By contrast,Pr⁡\[X3=Y\]=Pr⁡\[X4=Y\]=0\.8\\Pr\[X\_\{3\}=Y\]=\\Pr\[X\_\{4\}=Y\]=0\.8and a direct calculation gives

I​\(Y;X3,X4\)\\displaystyle I\(Y;X\_\{3\},X\_\{4\}\)=H​\(Y\)−H​\(Y∣X3,X4\)\\displaystyle=H\(Y\)\-H\(Y\\mid X\_\{3\},X\_\{4\}\)=1−\(0\.68⋅H​\(6468\)\+0\.32⋅H​\(12\)\)\>0,\\displaystyle=1\-\\left\(0\.68\\cdot H\\left\(\\tfrac\{64\}\{68\}\\right\)\+0\.32\\cdot H\\left\(\\tfrac\{1\}\{2\}\\right\)\\right\)\>0,whereH​\(⋅\)H\(\\cdot\)denotes binary entropy and the two terms correspond to the agreement and disagreement events for\(X3,X4\)\(X\_\{3\},X\_\{4\}\)\. Hence both accuracy\-first and mutual\-information greedy select\{3,4\}\\\{3,4\\\}\.

For diversity\-seeking, the pair\{1,2\}\\\{1,2\\\}has disagreementPr⁡\[X1≠X2\]=0\.1\\Pr\[X\_\{1\}\\neq X\_\{2\}\]=0\.1, which is strictly less thanPr⁡\[X3≠X4\]=0\.32\\Pr\[X\_\{3\}\\neq X\_\{4\}\]=0\.32\. Hence the diversity\-seeking never selects\{1,2\}\\\{1,2\\\}\. ∎

## Appendix BAdditional Results

### B\.1Label\-level Aggregation

In[Table4](https://arxiv.org/html/2605.24048#A2.T4),[5](https://arxiv.org/html/2605.24048#A2.T5), and[6](https://arxiv.org/html/2605.24048#A2.T6), we present the accuracy of each proposer while answering the questions independently\. As we can see, Aya is a weak model as a proposer, while we observe that it is a fast and accurate summarizer\.

Table 4:Independent accuracy for each proposer with rows indicating models, columns indicating prompt IDs on the AIME dataset\.Model12345GPT\-4o0\.3760\.380\.3520\.3590\.406AceReason\-Nemotron\-14B0\.3790\.3940\.3920\.3760\.372Llama\-3\.3\-70B\-Instruct0\.4560\.4660\.4570\.4440\.416QwQ\-32B0\.4160\.4220\.4100\.4390\.463Qwen3\-32B0\.4080\.4390\.4030\.420\.437Sky\-T1\-32B\-Preview0\.3870\.3980\.3810\.3830\.388aya\-expanse\-32b0\.2330\.2480\.2530\.2350\.267Gemini1\.5\-pro0\.50\.5040\.4590\.460\.488Table 5:Independent accuracy for each proposer with rows indicating models, columns indicating prompt IDs on the CLadder dataset\.Model12345GPT\-4o0\.6810\.6800\.6820\.6770\.685AceReason\-Nemotron\-14B0\.7080\.7260\.6920\.7070\.726Llama\-3\.3\-70B\-Instruct0\.5070\.5950\.5020\.5380\.532QwQ\-32B0\.7750\.7890\.8030\.8020\.797Qwen3\-32B0\.6780\.7170\.7100\.7070\.742Sky\-T1\-32B\-Preview0\.5990\.5870\.5910\.5830\.575aya\-expanse\-32b0\.5260\.5480\.5270\.5110\.519Gemini1\.5\-pro0\.7070\.7040\.7180\.7340\.748Table 6:Independent accuracy for each proposer with rows indicating models, columns indicating prompt IDs on the MMLU\-Pro dataset\.Model12345GPT\-4o0\.450\.4160\.4440\.450\.418AceReason\-Nemotron\-14B0\.5190\.5250\.5210\.5390\.507Llama\-3\.3\-70B\-Instruct0\.4730\.4630\.4440\.4540\.478QwQ\-32B0\.5820\.5890\.5730\.5760\.577Qwen3\-32B0\.640\.6370\.6220\.6160\.648Sky\-T1\-32B\-Preview0\.6510\.6460\.6440\.6420\.657aya\-expanse\-32b0\.3430\.3560\.3440\.2530\.368Gemini1\.5\-pro0\.5490\.530\.5360\.5370\.533We further test label\-level aggregators against LLM summarizer aggregation, aiming to show that leveraging proposers’ textual reasoning can boost accuracy\. We evaluate three majority\-vote variants: \(i\) over all proposers, \(ii\) over the best prompt per model, and \(iii\) over the best model per prompt\. We also include weighted majority vote, using the classic log\-odds weightswi∝log⁡pi1−piw\_\{i\}\\propto\\log\\frac\{p\_\{i\}\}\{1\-p\_\{i\}\}derived for independent binary voters byNitzan and Paroush \([1982](https://arxiv.org/html/2605.24048#bib.bib16)\)\. Although our setting involves multiclass labels and correlated voters, we adopt this weighting as a heuristic baseline\. Finally, we consider a learning\-based baseline that trains a decision tree on theNNproposers’ labels \(features\) to predict the ground truth, and report test accuracy\.

As shown in[Tables7](https://arxiv.org/html/2605.24048#A2.T7),[8](https://arxiv.org/html/2605.24048#A2.T8)and[9](https://arxiv.org/html/2605.24048#A2.T9), majority\-vote baselines perform competitively with the LLM summarizers on the binary CLadder dataset, yet they are outperformed by LLM summarizers on the multichoice AIME and MMLU\-Pro datasets with the best method\. These results underscore the importance of incorporating textual evidence from proposers’ reasoning, rather than relying solely on label\-level aggregation\.

Table 7:Label\-level aggregation baselines on AIME\.MethodAccuracyUnweightedWeightedMajority0\.6270\.644Majority \(best prompt per model\)0\.5830\.597Majority \(best model per prompt\)0\.5800\.580Decision Tree0\.488Table 8:Label\-level aggregation baselines on CLadder\.MethodAccuracyUnweightedWeightedMajority0\.7950\.814Majority \(best prompt per model\)0\.8160\.816Majority \(best model per prompt\)0\.7930\.805Decision Tree0\.737Table 9:Label\-level aggregation baselines on MMLU\-Pro\.MethodAccuracyUnweightedWeightedMajority0\.7470\.762Majority \(best prompt per model\)0\.7190\.740Majority \(best model per prompt\)0\.7360\.737Decision Tree0\.439
### B\.2Comparing Proposer Selection Methods \(Continued\)

Here, we present the comparison of methods in other settings, aiming to show the robustness of our methods\. We further present the composition of the selected proposer pool by each method to better illustrate their pros and cons\.

#### Ensemble Size

[Table11](https://arxiv.org/html/2605.24048#A2.T11)and[10](https://arxiv.org/html/2605.24048#A2.T10)report results for the \(AIME, complete pool, Aya,⋅\\cdot\) setting atk∈\{3,4\}k\\in\\\{3,4\\\}settings\. Note that the results for Input\-all, Best\-model, and MoA remain the same, as their performance does not depend onkk\. Our results confirm the robustness of our complementary\-MoA framework, as it remains competitive with the strongest baselines\. Overall, we do not observe a monotonic improvement in summarizer accuracy as the ensemble size increases\.

Table 10:A comparison of methods in the \(AIME, complete pool, Aya,k=3k=3\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.645Best\-model100–––––––0\.351Top\-accuracy6\.7–6\.786\.7––––0\.417MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.594Conditioned\-diversity––6\.733\.3––33\.326\.70\.416Aya\-dynamic–35\.352\.9––––11\.80\.414GPT5\.2\-dynamic–––40–––600\.470Truth\-prediction Greedy13\.333\.313\.333\.36\.7–––0\.714Oracle\-surrogate Greedy406\.72033\.3––––0\.528Model\-first Greedy53\.340–6\.7––––0\.710Table 11:A comparison of methods in the \(AIME, complete pool, Aya,k=4k=4\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.746Best\-model100–––––––0\.766Top\-accuracy7525––––––0\.740MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.660Conditioned\-diversity25–25–2525––0\.620Aya\-dynamic––3560–––50\.424GPT5\.2\-dynamic–––87\.1–––12\.90\.472Truth\-prediction Greedy5050––––––0\.740Oracle\-surrogate Greedy5025–25––––0\.728Model\-first Greedy2525–––––500\.796
#### Other Settings

Here, we present the results for the remaining settings\. Due to computational cost, we omit the LLM\-judge and Approximate Shapley baselines in some settings; where they were evaluated, neither outperformed our methods\. Two patterns emerge consistently across the tables\. First, our complementarity\-driven methods—particularly Model\-first Greedy—rank first or second in nearly every setting, regardless of dataset, summarizer, pool composition, or ensemble sizekk\. Second, the relative ordering of baselines is highly setting\-dependent: Best\-model is the strongest baseline for MMLU\-Pro with Aya yet among the worst on AIME, and Input\-all dominates AIME under Aya but underperforms under AceReason\. This sensitivity is precisely the failure mode our framework targets, and the consistent strength of complementarity\-aware selection across the same settings is the central empirical takeaway\.

Table 12:A comparison of methods in the \(AIME, complete pool, Aya,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.658Best\-model–––100––––0\.349Top\-accuracy8–3260––––0\.377MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.580Conditioned\-diversity––2020––40200\.483Aya\-dynamic–3260––––80\.435GPT5\.2\-dynamic–––64–––360\.448Approximate Shapley–––––––1000\.602Truth\-prediction Greedy28321220–––80\.611Oracle\-surrogate Greedy488–40–––40\.607Model\-first Greedy323244–––280\.654Table 13:A comparison of methods in the \(AIME, reduced pool, Aya,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all14\.314\.314\.314\.314\.314\.314\.30\.621Best\-model––100––––0\.332Top\-accuracy–4060––––0\.364MoA \(mixed\)14\.314\.314\.314\.314\.314\.314\.30\.562Conditioned\-diversity–2020––40200\.468Aya\-dynamic–3260–––80\.415GPT5\.2\-dynamic––59\.3––33\.37\.40\.450Approximate Shapley––––––1000\.596Truth\-prediction Greedy4032208–––0\.522Oracle\-surrogate Greedy202020–20–200\.497Model\-first Greedy3288–––520\.632Table 14:A comparison of methods in the \(AIME, complete pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.313Best\-model–––100––––0\.379Top\-accuracy8–3260––––0\.389MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.445Conditioned\-diversity––2020––40200\.476Aya\-dynamic–3260––––80\.436GPT5\.2\-dynamic–––64–––360\.487Truth\-prediction Greedy1236202012–––0\.465Oracle\-surrogate Greedy3681636–––40\.502Model\-first Greedy2812420––4320\.501Table 15:A comparison of methods in the \(AIME, reduced pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all14\.314\.314\.314\.314\.314\.314\.30\.344Best\-model––100––––0\.380Top\-accuracy–4060––––0\.397MoA14\.314\.314\.314\.314\.314\.314\.30\.441Conditioned\-diversity–2020––40200\.478Aya\-dynamic–3260–––80\.439GPT5\.2\-dynamic––59\.3––33\.37\.40\.472Truth\-prediction Greedy442428–––40\.466Oracle\-surrogate Greedy16876––––0\.406Model\-first Greedy24484424320\.498Table 16:A comparison of methods in the \(Cladder, complete pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.801Best\-model100–––––––0\.786Top\-accuracy100–––––––0\.777MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.757Conditioned\-diversity20–20–––60–0\.742Aya\-dynamic5248––––––0\.720GPT5\.2\-dynamic–40–60––––0\.760Truth\-prediction Greedy4020820–4–80\.762Oracle\-surrogate Greedy6012–28––––0\.752Model\-first Greedy–––6040–––0\.812Table 17:A comparison of methods in the \(Cladder, reduced pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all14\.314\.314\.314\.314\.314\.314\.30\.790Best\-model––80–––200\.793Top\-accuracy20–40–––400\.780MoA14\.314\.314\.314\.314\.314\.314\.30\.760Conditioned\-diversity82012––60–0\.765Aya\-dynamic88–––12––0\.748GPT5\.2\-dynamic40–60––––0\.763Truth\-prediction Greedy321228–––280\.761Oracle\-surrogate Greedy28–404––280\.765Model\-first Greedy––5644–––0\.802Table 18:A comparison of methods in the \(MMLU\-Pro, complete pool, Aya,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.377Best\-model–––––100––0\.524Top\-accuracy–36–––64––0\.495MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.448Conditioned\-diversity–820–201240–0\.397Aya\-dynamic–6036––4––0\.462GPT5\.2\-dynamic–––100––––0\.455Truth\-prediction Greedy840––4–48–0\.485Oracle\-surrogate Greedy456–––40––0\.487Model\-first Greedy52364–4––40\.502Table 19:A comparison of methods in the \(MMLU\-Pro, reduced pool, Aya,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTAyaAceReasonInput\-all14\.314\.314\.314\.314\.314\.314\.30\.346Best\-model–100–––––0\.482Top\-accuracy–100–––––0\.481MoA14\.314\.314\.314\.314\.314\.314\.30\.464Conditioned\-diversity–2020–2040–0\.404Aya\-dynamic–6436––––0\.460GPT5\.2\-dynamic–––100–––0\.455Truth\-prediction Greedy480–124––0\.478Oracle\-surrogate Greedy127648–––0\.487Model\-first Greedy324844––120\.475Table 20:A comparison of methods in the \(MMLU\-Pro, complete pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTSkyAyaAceReasonInput\-all12\.512\.512\.512\.512\.512\.512\.512\.50\.724Best\-model–––––100––0\.722Top\-accuracy–36–––64––0\.746MoA12\.512\.512\.512\.512\.512\.512\.512\.50\.737Conditioned\-diversity–820–201240–0\.683Aya\-dynamic–6036––4––0\.710GPT5\.2\-dynamic–––100––––0\.565Truth\-prediction Greedy4524–––40–0\.755Oracle\-surrogate Greedy2040–––40––0\.765Model\-first Greedy4016–128816–0\.738Table 21:A comparison of methods in the \(MMLU\-Pro, reduced pool, AceReason,k=5k=5\) setting\. Per column, the best accuracy is inbold blue, the second\-best is inblue, and the worst is inred\.MethodSelected proposers \(% of selections\)AccuracyQwQQwenLlamaGeminiGPTAyaAceReasonInput\-all14\.314\.314\.314\.314\.314\.314\.30\.687Best\-model–100–––––0\.664Top\-accuracy–100–––––0\.666MoA14\.314\.314\.314\.314\.314\.314\.30\.687Conditioned\-diversity–2020–2040–0\.669Aya\-dynamic–6436––––0\.692GPT5\.2\-dynamic–––100–––0\.568Truth\-prediction Greedy480–124––0\.678Oracle\-surrogate Greedy2064–124––0\.687Model\-first Greedy2824–161220–0\.711

## Appendix CPrompts

Multi\-choice — Proposer PromptYou will solve a multiple choice question\. Format your answer to include:1\.A full response2\.A concise step\-by\-step reasoning3\.The single letter choice

Binary\-choice — Proposer PromptYou will answer a yes or no question\. Format your answer to include:1\.A full response2\.A concise step\-by\-step reasoning3\.The yes or no answer

Multi\-choice — Summarizer PromptI will give you a multiple choice question and potential solutions that may be correct or incorrect\. Your task is to analyze the reasoning of the potential solutions step by step\.If there are any errors, correct them and update your answer\.If there are no errors, answer the question matching those solutions\.Your answer must be in the format of a full response, then a letter choice\.

Binary\-choice — Summarizer PromptI will give you a yes or no question and multiple potential solutions that may be correct or incorrect\. Your task is to analyze the reasoning of the potential solutions step by step\.If there are any errors, correct them and update your answer\.If there are no errors, answer the question matching those solutions\.Your answer must be in the format of a full response, then a yes or no answer\.

Instruction Prompt 1Divide the question into smaller, manageable parts and tackle each part individually before synthesizing the overall answer\.

Instruction Prompt 2Use mathematical principles and logic to solve the problem, even if it’s not a math question\.

Instruction Prompt 3Relate the question to a familiar concept or situation to better understand and solve it\.

Instruction Prompt 4Think about what the answer would be if the opposite were true, to gain a different perspective\.

Instruction Prompt 5Eliminate the obviously incorrect answers first and then choose the most likely correct answer\.

## Appendix DLLM Usage

Large language models \(LLMs\) were used in this paper only as a general\-purpose writing assistant\. Specifically, they supported adjusting phrasing for clarity, polishing grammar, shortening sentences, and reformatting text\. LLMs were also used to generate and refine tables \(e\.g\., aligning multi\-column headers and converting between LaTeX table styles\)\. At no point did LLMs contribute to research ideas, conceptual framing, or experimental design\. All substantive intellectual contributions are solely those of the authors\.