Competence-Aware Orchestration for LLM-Based Multi-Agent Systems

# Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
Source: [https://arxiv.org/html/2610.13144](https://arxiv.org/html/2610.13144)
Yuchen WangChenglin YuHongxia YangMing Li††thanks:Corresponding author:[ming\.li@polyu\.edu\.hk](mailto:[email protected])Affiliation:The Hong Kong Polytechnic University

###### Abstract

LLM\-based multi\-agent systems \(MAS\) have emerged as a promising paradigm for enhancing complex reasoning through role specialization and collaboration, yet a key challenge remains:*how can collaboration be adapted to a backbone’s competence on the current query?*This problem matters because more collaboration is not always better—richer coordination may help on difficult cases, but it also incurs additional token cost and redundant interactions\. We propose a two\-stage competence\-aware framework for automatic MAS orchestration\. It first models the LLM backbone’s task\-conditioned competence for the current query, and then uses this signal to adapt role activation and the communication graph\. Across six benchmarks, our method achieves an average score of 93\.81%\. Experiments with six LLM backbones of varying competence levels further show that our method adjusts collaboration according to the inferred competence: for higher\-competence backbones, it improves average accuracy from 87\.22 to 89\.97 while reducing token cost by 36\.26%; for lower\-competence backbones, it yields a further average gain of 4\.58 points over existing methods\. The same trend also holds on unseen backbones\.

## 1Introduction

Recent advances in large language models \(LLMs\) have greatly expanded the use of Multi\-Agent Systems \(MAS\) for reasoning, code generation, and complex problem solving\([Hong et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib7);[Wu et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib28);[Qian et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib18);[Chen et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib2)\)\. Related progress in prompting, search, self\-refinement, ensembling, and text\-based optimization has further improved LLM reasoning and adaptation\([Nori et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib16);[Shinn et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib22);[Jiang et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib9);[Yuksekgonul et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib33);[Liang et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib13)\)\. By assigning specialized roles and coordinating their interactions, LLM\-based MAS can outperform direct single\-agent reasoning\([Du et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib5);[Wang et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib25);[Liu et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib15)\)\. However, their effectiveness depends critically on collaboration topology and communication pattern: different tasks, and even different instances, may benefit from different forms and scales of collaboration\([Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36);[Yue et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib32);[Dang et al\., 2026](https://arxiv.org/html/2610.13144#bib.bib4)\)\. Moreover, more collaboration is not always better: richer role specialization and denser communication may help on difficult cases, but also introduce additional token cost, latency, and redundant interactions\([Zhang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib35);[Wang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib26)\)\. This motivates a competence\-aware perspective on automatic MAS construction:*How can collaboration be adapted to a backbone’s competence on the current query?*

![Refer to caption](https://arxiv.org/html/2610.13144v1/figs/Fig1_landscape.png)Figure 1:Comparison between existing MAS construction methods and our method\.Recent studies have moved from manually designed pipelines and hand\-crafted role\-based collaboration patterns\([Hong et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib7);[Wu et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib28);[Qian et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib18);[Du et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib5);[Wang et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib25)\)to frameworks that automatically optimize or dynamically generate multi\-agent structures\([Liu et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib15);[Zhuge et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib38);[Hu et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib8);[Zhang et al\., 2025d](https://arxiv.org/html/2610.13144#bib.bib37);[Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36);[Zhang et al\., 2025a](https://arxiv.org/html/2610.13144#bib.bib34);[Yue et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib32);[Dang et al\., 2026](https://arxiv.org/html/2610.13144#bib.bib4);[Wang et al\., 2025a](https://arxiv.org/html/2610.13144#bib.bib23)\)\. By generating role configurations and communication topologies from task descriptions, query content, candidate roles, or runtime feedback, these methods make MAS increasingly task\-aware and adaptive\([Zhang et al\., 2025d](https://arxiv.org/html/2610.13144#bib.bib37);[Dang et al\., 2026](https://arxiv.org/html/2610.13144#bib.bib4)\)\. Yet the main focus has been on organizing agent collaboration around task semantics, while relatively less attention has been paid to explicitly conditioning its scale and structure on the backbone’s competence for the current query\. We argue that the utility of collaboration is not fixed: it depends on the backbone’s standalone capability, the difficulty of the query, and the coordination overhead introduced by the system\. When the backbone’s standalone competence is high on a query, extra roles and intensive communication may add cost with little benefit\([Zhang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib35);[Wang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib26)\); in contrast, weaker backbones or harder queries may benefit substantially from richer coordination and verification\. This pattern also appears in broader agentic benchmarks\([Qian et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib19);[Kim et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib12)\)\. Motivated by these observations, we aim to adapt bothrole participationandcommunication structureto the backbone’stask\-conditioned competence\.

To this end, we propose a competence\-aware method for automatic MAS construction\. Instead of mapping a query directly to a collaboration structure, our method first constructs a task\-conditioned competence representation for the given backbone, and then uses it as a control signal for role activation and communication graph construction\.

Our contributions can be summarized as follows:

- •We design a competence modeling mechanism that combines a reusable backbone capability fingerprint with query\-specific task information to derive a backbone\-aware, query\-conditioned competence representation\.
- •We develop a competence\-aware orchestration method that uses this representation to configure the multi\-agent structure at two levels—role activation and communication graph generation—adapting team composition and coordination topology to both the backbone and the current query\.
- •Experiments across multiple benchmarks and backbones show that our method improves overall performance, reduces collaboration token usage on strong backbones, and brings larger gains on weaker backbones, with the same trend also observed on unseen backbones\.

## 2Related Work

##### Static multi\-agent collaboration and design\.

Early LLM\-based multi\-agent systems mainly relied on manually designed collaboration templates, including sequential pipelines and debate\-style communication[Du et al\. \(2024\)](https://arxiv.org/html/2610.13144#bib.bib5)\. Meanwhile, reasoning can also be improved within a single model through methods such as chain\-of\-thought[Wei et al\. \(2022\)](https://arxiv.org/html/2610.13144#bib.bib27)and self\-consistency[Wang et al\. \(2022\)](https://arxiv.org/html/2610.13144#bib.bib24)\. To reduce manual engineering in multi\-agent systems, subsequent work began to automate structure design, including workflow optimization methods such asAFlow[Zhang et al\. \(2025d\)](https://arxiv.org/html/2610.13144#bib.bib37)andDSPy[Khattab et al\. \(2024\)](https://arxiv.org/html/2610.13144#bib.bib11), as well as graph\-based approaches such asGPTSwarm[Zhuge et al\. \(2024\)](https://arxiv.org/html/2610.13144#bib.bib38),MacNet[Qian et al\. \(2025\)](https://arxiv.org/html/2610.13144#bib.bib19),AgentPrune[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.13144#bib.bib35), andAgentDropout[Wang et al\. \(2025b\)](https://arxiv.org/html/2610.13144#bib.bib26)\. These methods improve system design by searching, optimizing, or pruning collaboration structures before deployment\. However, whether hand\-crafted or automatically optimized, they remain largely input\-independent and follow a*learn once, deploy many*paradigm\.

##### Dynamic orchestration\.

More recent studies move beyond static structures and dynamically organize collaboration at inference time\. Rather than reusing one fixed design, these methods instantiate or route agent interaction according to the current query or runtime setting\. Representative examples includeG\-Designer, which generates query\-specific communication graphs[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.13144#bib.bib36);MasRouter, which routes topology, roles, and models[Yue et al\. \(2025\)](https://arxiv.org/html/2610.13144#bib.bib32); andMaAS,Puppeteer,CARD, andMaAS2, which dynamically orchestrate agent workflows under different settings[Zhang et al\. \(2025a\)](https://arxiv.org/html/2610.13144#bib.bib34);[Dang et al\. \(2026\)](https://arxiv.org/html/2610.13144#bib.bib4);[Wu et al\. \(2026\)](https://arxiv.org/html/2610.13144#bib.bib29);[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.13144#bib.bib23)\. This line of work makes multi\-agent systems more responsive to instance\-level variation\. However, existing methods mainly adapt collaboration to task semantics or runtime context, with limited explicit modeling of the backbone’s query\-specific competence\. Our work uses task\-conditioned competence as a signal for adapting role participation and communication\.

![Refer to caption](https://arxiv.org/html/2610.13144v1/figs/main_figure_v2.png)Figure 2:Overview of the proposed competence\-aware orchestration framework\. Stage I learns a competence representation from the backbone’s capability fingerprint and the current query under success\-rate supervision\. Stage II uses this representation to guide role activation and communication graph generation\.

## 3Method

### 3\.1Overview

Our method follows a two\-stage competence\-aware orchestration pipeline\. In Stage I, we learn a task\-conditioned competence representation from a reusable backbone capability fingerprint and query information\. In Stage II, we use this representation to adapt the multi\-agent structure at two levels: role activation and communication graph construction\.

### 3\.2Stage I: Competence Modeling

#### 3\.2\.1Capability Fingerprinting

To support downstream query\-level competence modeling, we precompute a reusable capability fingerprint for each LLM backbone using a fixed probe set𝒫\\mathcal\{P\}\. The probes are shared across backbones, organized by capability domain, and ordered within each domain by anchor\-calibrated difficulty, yielding a common easy\-to\-hard scale for direct comparison\. Note that this fingerprinting step is performed only once for each backbone: after a backboneMMis first evaluated on𝒫\\mathcal\{P\}, its fingerprintVmV\_\{m\}can be reused in all subsequent queries\.

We represent the fingerprint as a binary matrixVm∈\{0,1\}4×12V\_\{m\}\\in\\\{0,1\\\}^\{4\\times 12\}, where each row corresponds to one capability domain and the columns within each row are ordered from easy to hard\. Let𝒟=\{1,2,3,4\}\\mathcal\{D\}=\\\{1,2,3,4\\\}index the domains, and let\{xd,j\}j=112\\\{x\_\{d,j\}\\\}\_\{j=1\}^\{12\}denote the fixed probes in domaindd\. RunningMMon each probexd,jx\_\{d,j\}yields a binary correctness signalr⁡\(xd,j,M\)∈\{0,1\}r\(x\_\{d,j\};M\)\\in\\\{0,1\\\}, from which we construct

Vm​\[d,j\]=r⁡\(xd,j,M\),d∈𝒟\.V\_\{m\}\[d,j\]\\;=\\;r\(x\_\{d,j\};M\),\\quad d\\in\\mathcal\{D\}\.\(1\)Because probe identities and their within\-domain difficulty ordering are shared across backbones,VmV\_\{m\}serves as a directly comparable capability prior for downstream competence modeling: each row summarizes performance in one capability domain, while the column position reflects how far competence extends along the shared difficulty scale\. Full probe\-set construction details are provided in Appendix[A\.1](https://arxiv.org/html/2610.13144#A1.SS1)\.

#### 3\.2\.2Task\-Conditioned Competence Modeling

To condition the capability priorVmV\_\{\\mathrm\{m\}\}on the current query, we combine it with the task representation to learn a task\-conditioned competence representationhtcch\_\{\\mathrm\{tcc\}\}\. Query\-level success rates provide proxy supervision for learning this representation, which then conditions downstream role activation and communication graph construction\.

Given a queryqq, we first encode it into a task representationVt∈ℝdV\_\{\\mathrm\{t\}\}\\in\\mathbb\{R\}^\{d\}\. To capture the domain demands implied byqq, we introduce a set of learnable domain prototypesVd∈ℝ4×dV\_\{\\mathrm\{d\}\}\\in\\mathbb\{R\}^\{4\\times d\}, with one prototype for each capability domain inVmV\_\{\\mathrm\{m\}\}, initialized from the embeddings of short GPT5\-generated domain descriptions\. We then compute the similarity betweenVtV\_\{\\mathrm\{t\}\}and each domain prototype, and normalize the resulting scores with a softmax to obtain domain relevance weightsw∈ℝ4w\\in\\mathbb\{R\}^\{4\}\. Usingww, we aggregate the domain\-wise fingerprint into task\-conditioned capability evidence:

Vm′=∑i=14wi⋅Vm\[i,:\]\.V\_\{\\mathrm\{m\}\}^\{\\prime\}=\\sum\_\{i=1\}^\{4\}w\_\{i\}\\cdot V\_\{\\mathrm\{m\}\}\[i,:\]\.\(2\)Since the columns ofVmV\_\{\\mathrm\{m\}\}are ordered by increasing difficulty,Vm′V\_\{\\mathrm\{m\}\}^\{\\prime\}provides a query\-relevant and difficulty\-aware summary of the backbone’s capability fingerprint, which is then fused with task semantics to form the final competence representation\.

We then projectVtV\_\{\\mathrm\{t\}\}andVm′V\_\{\\mathrm\{m\}\}^\{\\prime\}into a shared latent space, denoted byhth\_\{\\mathrm\{t\}\}andhmh\_\{\\mathrm\{m\}\}, and fuse them through an interaction\-aware fusion layer:

htcc=ϕf​\(\[ht;hm;\(ht−hm\);\(ht⊙hm\)\]\),h\_\{\\mathrm\{tcc\}\}=\\phi\_\{\\mathrm\{f\}\}\\\!\\Big\(\\big\[\\,h\_\{\\mathrm\{t\}\};\\ h\_\{\\mathrm\{m\}\};\\ \(h\_\{\\mathrm\{t\}\}\-h\_\{\\mathrm\{m\}\}\);\\ \(h\_\{\\mathrm\{t\}\}\\odot h\_\{\\mathrm\{m\}\}\)\\,\\big\]\\Big\),\(3\)where⊙\\odotdenotes element\-wise multiplication,\[⋅;⋅\]\[\\cdot;\\cdot\]denotes concatenation, andϕf​\(⋅\)\\phi\_\{\\mathrm\{f\}\}\(\\cdot\)is a learnable fusion network\.

We use query\-level success rates as proxy supervision forhtcch\_\{\\mathrm\{tcc\}\}\. An auxiliary linear head followed by a sigmoid maps this representation to a predicted success rate:

p^=σ⁡\(Proxy⁡\(htcc\)\)\.\\hat\{p\}=\\sigma\\big\(\\mathrm\{Proxy\}\(h\_\{\\mathrm\{tcc\}\}\)\\big\)\.\(4\)The supervision targety∈\[0,1\]y\\in\[0,1\]is the empirical success rate of the corresponding LLM backbone on the query, estimated from repeated trials\. Details of the training data construction are provided in Appendix[A\.2](https://arxiv.org/html/2610.13144#A1.SS2)\. We optimize the model by minimizing the binary cross\-entropy \(BCE\) between the predicted success ratep^\\hat\{p\}and the supervision targetyy\. The prediction head supplies auxiliary supervision for learning competence\-related information; downstream role activation and communication graph generation are conditioned on the representationhtcch\_\{\\mathrm\{tcc\}\}\.

### 3\.3Stage II: Competence\-Aware MAS Orchestration

Given the task\-conditioned competence representation produced in Stage I, Stage II adapts the multi\-agent structure for the current query through two mechanisms: role activation and communication graph construction\. The former assesses how suitable each role is for participation, while the latter determines how the activated roles should communicate\.

#### 3\.3\.1Role Activation

Given the query representationVtV\_\{\\mathrm\{t\}\}and the task\-conditioned competence representationhtcch\_\{\\mathrm\{tcc\}\}, role activation predicts a query\-specific activation vectorA∈\(0,1\)NA\\in\(0,1\)^\{N\}over the candidate role pool\. It combines task–role relevance with competence\-conditioned modulation, and the resultingAAserves as a prior for downstream communication graph construction\.

Let the candidate role pool beℛrole=\{ri\}i=1N\\mathcal\{R\}\_\{\\mathrm\{role\}\}=\\\{r\_\{i\}\\\}\_\{i=1\}^\{N\}, where each rolerir\_\{i\}is associated with a textual descriptiondid\_\{i\}\. We encode each description into a role representationhi∈ℝdrh\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{r\}\}\}, and compute a relevance gate based on projected task–role similarity to modulate each role representation:

gi=σ⁡\(\(Wt​Vt\)⊤​\(Wr​hi\)\),hi′=gi⋅hi\.g\_\{i\}=\\sigma\\\!\\left\(\(W\_\{\\mathrm\{t\}\}V\_\{\\mathrm\{t\}\}\)^\{\\top\}\(W\_\{\\mathrm\{r\}\}h\_\{i\}\)\\right\),\\quad h^\{\\prime\}\_\{i\}=g\_\{i\}\\cdot h\_\{i\}\.\(5\)We then predict the final activation score of roleiiby conditioning on both the gated role representation and the task\-conditioned competence signal:

Ai=σ\(MLP\(\[hi′;htcc\]\)\),i=1,…,N,A\_\{i\}=\\sigma\\\!\\Big\(\\mathrm\{MLP\}\\big\(\[h^\{\\prime\}\_\{i\};\\,h\_\{\\mathrm\{tcc\}\}\]\\big\)\\Big\),\\quad i=1,\\dots,N,\(6\)yielding𝐀=\[A1,…,AN\]∈\(0,1\)N\\mathbf\{A\}=\[A\_\{1\},\\dots,A\_\{N\}\]\\in\(0,1\)^\{N\}\. Here, task–role relevance determines which roles are plausible for the current query, whilehtcch\_\{\\mathrm\{tcc\}\}modulates how strongly they should be activated given the backbone’s estimated competence\.

Since direct supervision for role activation is unavailable, we train the activator with two weak supervision signals\. The first is a semantic anchor derived from task–role similarity, which provides role\-level relevance supervision\. The second reuses the query\-level success\-rate supervision from Section[3\.2\.2](https://arxiv.org/html/2610.13144#S3.SS2.SSS2)to form a soft budget prior that regulates the overall activation scale\.

Specifically, we compute the cosine similarity between the projected task representation and each projected role representation:

V~t=Wt​Vt,h~i=Wr​hi,si=cos⁡\(V~t,h~i\)\.\\tilde\{V\}\_\{\\mathrm\{t\}\}=W\_\{\\mathrm\{t\}\}V\_\{\\mathrm\{t\}\},\\;\\tilde\{h\}\_\{i\}=W\_\{\\mathrm\{r\}\}h\_\{i\},\\;s\_\{i\}=\\cos\(\\tilde\{V\}\_\{\\mathrm\{t\}\},\\tilde\{h\}\_\{i\}\)\.\(7\)We then apply min–max normalization over the role pool to obtain a query\-local semantic anchor targetPanchorP\_\{\\text\{anchor\}\}\. LetPanchor=\[Panchor,1,…,Panchor,N\]∈\[0,1\]NP\_\{\\text\{anchor\}\}=\[P\_\{\\text\{anchor\},1\},\\dots,P\_\{\\text\{anchor\},N\}\]\\in\[0,1\]^\{N\}\. We align the predicted activationsAAwith this semantic anchor using

ℒanchor=MSE⁡\(𝐀,Panchor\)\.\\mathcal\{L\}\_\{\\text\{anchor\}\}=\\operatorname\{MSE\}\\\!\\big\(\\mathbf\{A\},P\_\{\\text\{anchor\}\}\\big\)\.\(8\)This objective provides relative supervision within each query, encouraging higher activation for roles that are more semantically compatible with the task\.

To regulate the overall activation scale, we again reuse the success\-rate supervision targety∈\[0,1\]y\\in\[0,1\]from Section[3\.2\.2](https://arxiv.org/html/2610.13144#S3.SS2.SSS2)\. We use1−y1\-yas a query\-level soft budget prior and regularize the mean role activation toward this target:

ℒbudget=\(mean⁡\(𝐀\)−\(1−y\)\)2\.\\mathcal\{L\}\_\{\\text\{budget\}\}=\\Big\(\\operatorname\{mean\}\(\\mathbf\{A\}\)\-\(1\-y\)\\Big\)^\{2\}\.\(9\)This term encourages lower mean activation for queries with higher standalone success rates and higher mean activation for those with lower success rates\. Together with the semantic anchor, it provides weak supervision for competence\-conditioned role selection\.

The overall objective is

ℒ=λanchor​ℒanchor\+λbudget​ℒbudget\.\\mathcal\{L\}=\\lambda\_\{\\text\{anchor\}\}\\mathcal\{L\}\_\{\\text\{anchor\}\}\+\\lambda\_\{\\text\{budget\}\}\\mathcal\{L\}\_\{\\text\{budget\}\}\.\(10\)The resulting activation vectorAAdetermines role participation for the current query and serves as input to downstream communication topology generation\.

#### 3\.3\.2Graph\-based MAS Generation

Building on the competence\-aware role activation above, we generate a directed communication graph for the current query\. Specifically, given the task representationVtV\_\{\\mathrm\{t\}\}, the task\-conditioned competence representationhtcch\_\{\\mathrm\{tcc\}\}, the competence\-aware role representations𝐇′=\{hi′\}i=1N\\mathbf\{H\}^\{\\prime\}=\\\{h\_\{i\}^\{\\prime\}\\\}\_\{i=1\}^\{N\}, and the corresponding activation scores𝐀=\{Ai\}i=1N\\mathbf\{A\}=\\\{A\_\{i\}\\\}\_\{i=1\}^\{N\}, we predict a directed communication graphGcom=\(𝒱,ℰcom\)G\_\{\\mathrm\{com\}\}=\(\\mathcal\{V\},\\mathcal\{E\}\_\{\\mathrm\{com\}\}\)over the role set𝒱=\{1,…,N\}\\mathcal\{V\}=\\\{1,\\dots,N\\\}\. Each directed edge represents a potential information\-flow path used in multi\-round multi\-agent interaction\. Similar to[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.13144#bib.bib36), we initialize the graph encoder with an anchor topology, which serves only as a structural prior for message passing\.

##### Task\-conditioned graph encoder\.

To predict the query\-specific communication topology, we encode each role node using its competence\-aware role representation and activation score\. Specifically, we use the activation scoreAiA\_\{i\}to modulate the corresponding role representation:

h¯i=hi′⊙\(1\+α​Ai\),\\bar\{h\}\_\{i\}=h\_\{i\}^\{\\prime\}\\odot\(1\+\\alpha A\_\{i\}\),\(11\)whereα\\alphais a scaling coefficient\. We then construct the input feature of each role node by concatenating the modulated role representation with the task representationVtV\_\{\\mathrm\{t\}\}and the task\-conditioned competence contexthtcch\_\{\\mathrm\{tcc\}\}:

xi=\[h¯i;Vt;htcc\]\.x\_\{i\}=\[\\,\\bar\{h\}\_\{i\};\\,V\_\{\\mathrm\{t\}\};\\,h\_\{\\mathrm\{tcc\}\}\\,\]\.\(12\)Let𝐗=\{xi\}i=1N\\mathbf\{X\}=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}denote the resulting node features\. A graph encoderϕθ\\phi\_\{\\theta\}then performs message passing over the initialized role graph and produces contextualized node representations:

𝐇~=ϕθ​\(𝐗\),\\tilde\{\\mathbf\{H\}\}=\\phi\_\{\\theta\}\(\\mathbf\{X\}\),\(13\)where𝐇~=\{h~i\}i=1N\\tilde\{\\mathbf\{H\}\}=\\\{\\tilde\{h\}\_\{i\}\\\}\_\{i=1\}^\{N\}\. In this way, topology prediction is jointly conditioned on role semantics, task information, task\-conditioned competence context, and role activation strength\.

##### Edge decoding and graph sampling\.

Based on the encoded node representations, we compute an edge score for each allowable directed edge\(i,j\)∈ℰallow\(i,j\)\\in\\mathcal\{E\}\_\{\\mathrm\{allow\}\}:

Si​j=σ⁡\(fedge​\(h~i,h~j\)\),S\_\{ij\}=\\sigma\\\!\\bigl\(f\_\{\\mathrm\{edge\}\}\(\\tilde\{h\}\_\{i\},\\tilde\{h\}\_\{j\}\)\\bigr\),\(14\)wherefedgef\_\{\\mathrm\{edge\}\}is the edge decoder andSi​j∈\(0,1\)S\_\{ij\}\\in\(0,1\)is interpreted as the activation probability of edge\(i,j\)\(i,j\)\. The query\-specific communication graphGcomG\_\{\\mathrm\{com\}\}is obtained by selecting edges according to these scores over the allowable edge set\. During training, we sample a graph instanceG^\\hat\{G\}from the predicted edge distribution for rollout and optimization\. We optimize the expected utility–cost trade\-off\. Specifically, the utility term is defined as

ℒutil=−𝔼G^∼pθ\(⋅∣𝐇′,𝐀,Vt,ht​c​c\)\[u\(G^\)\],\\mathcal\{L\}\_\{\\text\{util\}\}=\-\\mathbb\{E\}\_\{\\hat\{G\}\\sim p\_\{\\theta\}\(\\cdot\\mid\\mathbf\{H\}^\{\\prime\},\\mathbf\{A\},V\_\{t\},h\_\{tcc\}\)\}\\big\[u\(\\hat\{G\}\)\\big\],\(15\)whereu⁡\(G^\)u\(\\hat\{G\}\)denotes the utility obtained from the sampled topologyG^\\hat\{G\}\. The communication cost term is defined as

ℒcost=ρ⁡\(y\)​𝒞​\(S\),\\mathcal\{L\}\_\{\\text\{cost\}\}=\\rho\(y\)\\,\\mathcal\{C\}\(S\),\(16\)wherey∈\[0,1\]y\\in\[0,1\]is the instance\-wise success\-rate signal introduced in Section[3\.2\.2](https://arxiv.org/html/2610.13144#S3.SS2.SSS2), namely, the empirical success rate of the base model on the current query estimated from repeated trials\. We use this signal to modulate the communication penalty and define the communication cost as

ρ⁡\(y\)=max⁡\(y,yϵ\),𝒞⁡\(S\)=∑\(i,j\)∈ℰallowSi​j\.\\rho\(y\)=\\max\(y,y\_\{\\epsilon\}\),\\;\\mathcal\{C\}\(S\)=\\sum\_\{\(i,j\)\\in\\mathcal\{E\}\_\{\\mathrm\{allow\}\}\}S\_\{ij\}\.\(17\)Hereyϵ\>0y\_\{\\epsilon\}\>0is a small floor constant, and𝒞⁡\(S\)\\mathcal\{C\}\(S\)corresponds to the expected number of active edges under the predicted edge probabilities\. This competence\-conditioned weight reduces the relative communication penalty on queries with lower standalone success rates, while the utility term rewards task performance\. The overall objective is

ℒgraph​\(θ\)=ℒutil\+λ​ℒcost,\\mathcal\{L\}\_\{\\mathrm\{graph\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{util\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{cost\}\},\(18\)whereλ\\lambdabalances task utility and communication cost\. The utility term is optimized with REINFORCE, while the cost term backpropagates directly throughSS\.

## 4Experiments

### 4\.1Experimental Setting

##### Datasets and Metrics\.

Following prior work on automatic multi\-agent system design and orchestration\([Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36)\), we evaluate our method on six widely used benchmark datasets: MMLU\([Hendrycks et al\., 2020](https://arxiv.org/html/2610.13144#bib.bib6)\), GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.13144#bib.bib3)\), MultiArith\([Roy & Roth, 2015](https://arxiv.org/html/2610.13144#bib.bib21)\), SVAMP\([Patel et al\., 2021](https://arxiv.org/html/2610.13144#bib.bib17)\), AQuA\([Ling et al\., 2017](https://arxiv.org/html/2610.13144#bib.bib14)\), and HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.13144#bib.bib1)\)\. We report the standard evaluation metric for each dataset, using accuracy for MMLU, GSM8K, MultiArith, SVAMP, and AQuA, and Pass@1 for HumanEval\. Dataset statistics are reported in Table[6](https://arxiv.org/html/2610.13144#A1.T6)\.

##### Baselines\.

Following prior work\([Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36)\), we compare our method with three groups of baselines\. The first group consists of*single\-agent methods*, including Chain\-of\-Thought \(CoT\)\([Wei et al\., 2022](https://arxiv.org/html/2610.13144#bib.bib27)\)and Self\-Consistency\([Wang et al\., 2022](https://arxiv.org/html/2610.13144#bib.bib24)\)\. The second group includes*multi\-agent systems with fixed collaboration patterns*, such as Chain, Tree, Complete Graph, and Random Graph\([Qian et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib19)\), as well as debate\-based interaction instantiated by LLM\-Debate\([Du et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib5)\)\. The third group covers*multi\-agent methods with dynamic or optimized structures*, including AgentPrune\([Zhang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib35)\), AgentDropout\([Wang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib26)\), MasRouter\([Yue et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib32)\), CARD\([Wu et al\., 2026](https://arxiv.org/html/2610.13144#bib.bib29)\), and G\-Designer\([Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36)\)\.

##### Implementation Details

We access GPT models via the OpenAI API, primarily usinggpt\-4o\-2024\-08\-06\(GPT\-4o\) as the target backbone\. Unless otherwise specified, all LLM calls in evaluation use a decoding temperature of 0\. For multi\-agent methods, we adopt explicit agent profiling with a predefined pool of textual role descriptions, following classical configurations in prior LLM\-based MAS work\([Liu et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib15);[Zhuge et al\., 2024](https://arxiv.org/html/2610.13144#bib.bib38);[Yin et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib31)\)\. We additionally include a summarizer agent to aggregate the dialogue history and produce the final answer\. Both task queries and role descriptions are encoded usingbge\-small\-en\-v1\.5\([Xiao et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib30)\), yielding 384\-dimensional embeddings\. We train Task\-Conditioned Competence Modeling, Role Activation, and Graph\-based MAS Generation sequentially offline, keeping the other modules frozen at each stage\. Deployment reuses the trained model without further parameter updates\. For training the graph\-generation module, following[Zhang et al\. \(2025c\)](https://arxiv.org/html/2610.13144#bib.bib36), we use 40 queries from each dataset for training\. We use dataset\-specific training and evaluation splits; their definitions and the construction of capability fingerprints and supervision targets are detailed in Appendix[A](https://arxiv.org/html/2610.13144#A1)\.

### 4\.2Performance Comparison

The comparison results are presented in Table[1](https://arxiv.org/html/2610.13144#S4.T1)\. Under the same backbone setting as all baselines, our method achieves the best average performance of 93\.81% across the six benchmarks, establishing state\-of\-the\-art overall results\. In Section[4\.4](https://arxiv.org/html/2610.13144#S4.SS4), we further examine whether this advantage generalizes across different target backbones, including unseen models\.

Table 1:Performance comparison \(%\) on six benchmarks\.
### 4\.3Ablation Study

Our full method learns its competence representation from two inputs: the task representationVtV\_\{\\mathrm\{t\}\}and the capability fingerprintVmV\_\{\\mathrm\{m\}\}\. We evaluate their downstream contribution through three ablated variants\.w/o taskremovesVtV\_\{\\mathrm\{t\}\}from the competence module, so its representation is no longer query\-conditioned\.w/o modelremovesVmV\_\{\\mathrm\{m\}\}, so the module no longer has access to a model\-specific capability prior\.w/ desc\.replacesVmV\_\{\\mathrm\{m\}\}with free\-form model descriptions \(e\.g\., textual summaries of parameter scale and characteristics\)\.

Table 2:Downstream ablation of competence modeling\.Task conditioning improves the performance–cost trade\-off\.Removing the task representation reduces the average score by 2\.55 points and increases token cost by 16\.84%\. These results support conditioning the competence representation on the current query for downstream orchestration in this setting\.

The capability prior helps control collaboration cost\.RemovingVmV\_\{\\mathrm\{m\}\}causes the largest token increase \(\+154\.64%\) and also lowers the average score by 3\.02 points\. Under our orchestration design, the backbone\-specific capability prior thus supports lower token usage together with higher average performance\.

Structured fingerprints outperform the evaluated textual descriptions\.ReplacingVmV\_\{\\mathrm\{m\}\}with textual descriptions drops the average score by 3\.10 points and increases token cost by 100\.67%\. This comparison favors the structured fingerprint over these descriptions as a conditioning signal for orchestration\.

### 4\.4Cross\-Backbone Performance and Generalization to Unseen Models

We evaluate our competence\-aware collaboration on MMLU with performance\-oriented and efficiency\-oriented backbones\. We mark as unseen \(⋆\) backbones used neither as probe\-construction anchors nor for competence\-supervision data collection\. Tables[3](https://arxiv.org/html/2610.13144#S4.T3)and[4](https://arxiv.org/html/2610.13144#S4.T4)report accuracy, with Avg\. computed across the three backbones in each table\. Figure[3](https://arxiv.org/html/2610.13144#S4.F3)shows agent counts across backbones and MMLU queries\.

![Refer to caption](https://arxiv.org/html/2610.13144v1/figs/activation.png)Figure 3:Number of agents in the generated MAS on efficiency\-oriented target backbones\.Table 3:Results on MMLU with performance\-oriented target backbones\.On performance\-oriented backbones, our method maintains strong performance with substantially lower collaboration cost\.As shown in Table[3](https://arxiv.org/html/2610.13144#S4.T3), our method achieves the best average score on this group, improving over G\-Designer from 87\.22 to 89\.97, while using↓\\downarrow36\.26%fewer tokens per query after deployment\. Capability fingerprinting adds a small one\-time cost per backbone, which these savings recover after only about 28 queries on average across this group \(Appendix[A\.3](https://arxiv.org/html/2610.13144#A1.SS3)\)\. This result shows that performance\-oriented backbones can still benefit from our collaboration strategy, while incurring much lower collaboration overhead\.

Table 4:Results on MMLU with efficiency\-oriented target backbones\.On efficiency\-oriented backbones, our method brings larger performance gains\.In Table[4](https://arxiv.org/html/2610.13144#S4.T4), our method improves the average score from 74\.07 to 78\.65, with the largest gain on Gemma\-3\-12B\. Figure[3](https://arxiv.org/html/2610.13144#S4.F3)further shows that the number of agents in the generated MAS varies substantially across target models and tasks\.

These improvements also hold on unseen LLM backbones\.The gains also hold on unseen backbones GLM 5\.1⋆and Gemini 3\.1 Pro⋆\. These results support transferring competence\-conditioned orchestration beyond the backbones used for probe construction and supervision\. We further evaluate the trained model on unseen datasets with an additional unseen backbone in Appendix[A\.4](https://arxiv.org/html/2610.13144#A1.SS4)\.

### 4\.5Case Study

We present two MMLU case studies in Figure[4](https://arxiv.org/html/2610.13144#S4.F4)to show how competence\-aware orchestration adapts MAS to different backbone–task settings\.

##### Changing the task while fixing the backbone\.

We fix the backbone to Llama\-3\.1\-8B and compare two psychology questions fromhigh\_school\_psychologyandprofessional\_psychology\. The latter requires more specialized expertise\. Without explicit competence\-aware adaptation, G\-Designer makes only limited structural changes and fails on the harder question\. In this example, our method activates additional roles and returns a correct answer, illustrating query\-dependent adaptation of the collaboration structure\.

##### Changing the backbone while fixing the task\.

We then fix the question and change the backbone from Llama\-3\.1\-8B to GPT\-4o\. G\-Designer continues to instantiate a relatively large collaboration structure\. In contrast, our method activates fewer roles with GPT\-4o while preserving the correct answer and reducing token cost by 42\.2%\. This example illustrates backbone\-dependent adaptation with lower collaboration cost\.

![Refer to caption](https://arxiv.org/html/2610.13144v1/figs/case_study.png)Figure 4:Case studies on MMLU\. Left: changing the task while fixing the backbone \(Llama\-3\.1\-8B\)\. Right: changing the backbone from Llama\-3\.1\-8B to GPT\-4o while fixing the task\. check/cross marks indicate answer correctness\.

## 5Conclusion

In this paper, we study*how to adapt multi\-agent collaboration to a backbone’s competence on the current query*\. We propose a competence\-aware framework that learns a task\-conditioned representation under success\-rate supervision and uses it to guide role activation and communication topology\. Experiments across six benchmarks and multiple LLM backbones show that our method improves performance, reduces collaboration cost on stronger backbones, and yields larger gains on weaker ones, with the same trend also observed on unseen backbones\.

### AI use statement

Generative AI tools supported this work, including language polishing and code implementation and debugging\. The authors reviewed the AI\-assisted materials and take full responsibility for the final manuscript and results\.

## References

- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Brockman\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- Chen et al\. \(2024\)Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi\-Min Chan, Heyang Yu, Yaxi Lu, Yi\-Hsin Hung, Chen Qian, et al\.Agentverse: Facilitating multi\-agent collaboration and exploring emergent behaviors\.In*International Conference on Learning Representations*, volume 2024, pp\. 20094–20136, 2024\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Dang et al\. \(2026\)Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al\.Multi\-agent collaboration via evolving orchestration\.*Advances in neural information processing systems*, 38:165025–165059, 2026\.
- Du et al\. \(2024\)Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch\.Improving factuality and reasoning in language models through multiagent debate\.In*Forty\-first international conference on machine learning*, 2024\.
- Hendrycks et al\. \(2020\)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*, 2020\.
- Hong et al\. \(2024\)Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Steven Yau, Zijuan Lin, Liyang Zhou, et al\.Metagpt: Meta programming for a multi\-agent collaborative framework\.In*International Conference on Learning Representations*, volume 2024, pp\. 23247–23275, 2024\.
- Hu et al\. \(2025\)Shengran Hu, Cong Lu, and Jeff Clune\.Automated design of agentic systems\.In*International Conference on Learning Representations*, volume 2025, pp\. 21344–21377, 2025\.
- Jiang et al\. \(2023\)Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin\.Llm\-blender: Ensembling large language models with pairwise ranking and generative fusion\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 14165–14178, 2023\.
- Kazemi et al\. \(2025\)Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K\. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q\. Tran, Quoc V\. Le, and Orhan Firat\.BIG\-bench extra hard\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 26473–26501, 2025\.doi:10\.18653/v1/2025\.acl\-long\.1285\.
- Khattab et al\. \(2024\)Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, et al\.Dspy: compiling declarative language model calls into state\-of\-the\-art pipelines\.In*International Conference on Learning Representations*, volume 2024, pp\. 54928–54958, 2024\.
- Kim et al\. \(2025\)Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, et al\.Towards a science of scaling agent systems\.*arXiv preprint arXiv:2512\.08296*, 2025\.
- Liang et al\. \(2024\)Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu\.Encouraging divergent thinking in large language models through multi\-agent debate\.In*Proceedings of the 2024 conference on empirical methods in natural language processing*, pp\. 17889–17904, 2024\.
- Ling et al\. \(2017\)Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom\.Program induction by rationale generation: Learning to solve and explain algebraic word problems\.In*Proceedings of the 55th annual meeting of the association for computational linguistics \(volume 1: Long papers\)*, pp\. 158–167, 2017\.
- Liu et al\. \(2023\)Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang\.Dynamic llm\-agent network: An llm\-agent collaboration framework with agent team optimization\.*arXiv preprint arXiv:2310\.02170*, 2023\.
- Nori et al\. \(2023\)Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al\.Can generalist foundation models outcompete special\-purpose tuning? case study in medicine\.*arXiv preprint arXiv:2311\.16452*, 2023\.
- Patel et al\. \(2021\)Arkil Patel, Satwik Bhattamishra, and Navin Goyal\.Are nlp models really able to solve simple math word problems?In*Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies*, pp\. 2080–2094, 2021\.
- Qian et al\. \(2024\)Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al\.Chatdev: Communicative agents for software development\.In*Proceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\)*, pp\. 15174–15186, 2024\.
- Qian et al\. \(2025\)Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al\.Scaling large language model\-based multi\-agent collaboration\.In*International Conference on Learning Representations*, volume 2025, pp\. 41488–41505, 2025\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.GPQA: A graduate\-level google\-proof Q&A benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.
- Roy & Roth \(2015\)Subhro Roy and Dan Roth\.Solving general arithmetic word problems\.In*Proceedings of the 2015 conference on empirical methods in natural language processing*, pp\. 1743–1752, 2015\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.*Advances in neural information processing systems*, 36:8634–8652, 2023\.
- Wang et al\. \(2025a\)Kun Wang, Guibin Zhang, ManKit Ye, Xinyu Deng, Dongxia Wang, Xiaobin Hu, Jinyang Guo, Yang Liu, and Yufei Guo\.Mas2: Self\-generative, self\-configuring, self\-rectifying multi\-agent systems\.*arXiv preprint arXiv:2509\.24323*, 2025a\.
- Wang et al\. \(2022\)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou\.Self\-consistency improves chain of thought reasoning in language models\.*arXiv preprint arXiv:2203\.11171*, 2022\.
- Wang et al\. \(2024\)Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji\.Unleashing the emergent cognitive synergy in large language models: A task\-solving agent through multi\-persona self\-collaboration\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 257–279, 2024\.
- Wang et al\. \(2025b\)Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang\.Agentdropout: Dynamic agent elimination for token\-efficient and high\-performance llm\-based multi\-agent collaboration\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 24013–24035, 2025b\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.*Advances in neural information processing systems*, 35:24824–24837, 2022\.
- Wu et al\. \(2024\)Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al\.Autogen: Enabling next\-gen llm applications via multi\-agent conversations\.In*First conference on language modeling*, 2024\.
- Wu et al\. \(2026\)Tongtong Wu, Yanming Li, Ziye Tang, Chen Jiang, Linhao Luo, Guilin Qi, Shirui Pan, and Gholamreza Haffari\.Card: Towards conditional design of multi\-agent topological structures\.In*The Fourteenth International Conference on Learning Representations*, 2026\.
- Xiao et al\. \(2023\)Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff\.C\-pack: Packaged resources to advance general chinese embedding, 2023\.
- Yin et al\. \(2023\)Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuan\-Jing Huang, and Xipeng Qiu\.Exchange\-of\-thought: Enhancing large language model capabilities through cross\-model communication\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp\. 15135–15153, 2023\.
- Yue et al\. \(2025\)Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi\.Masrouter: Learning to route llms for multi\-agent systems\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 15549–15572, 2025\.
- Yuksekgonul et al\. \(2024\)Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou\.Textgrad: Automatic" differentiation" via text\.*arXiv preprint arXiv:2406\.07496*, 2024\.
- Zhang et al\. \(2025a\)Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang\.Multi\-agent architecture search via agentic supernet\.In*International Conference on Machine Learning*, pp\. 75834–75852\. PMLR, 2025a\.
- Zhang et al\. \(2025b\)Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Yu, and Tianlong Chen\.Cut the crap: An economical communication pipeline for llm\-based multi\-agent systems\.In*International Conference on Learning Representations*, volume 2025, pp\. 75389–75428, 2025b\.
- Zhang et al\. \(2025c\)Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng\.G\-designer: Architecting multi\-agent communication topologies via graph neural networks\.In*International Conference on Machine Learning*, pp\. 76678–76692\. PMLR, 2025c\.
- Zhang et al\. \(2025d\)Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al\.Aflow: Automating agentic workflow generation\.In*International Conference on Learning Representations*, volume 2025, pp\. 34040–34077, 2025d\.
- Zhuge et al\. \(2024\)Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber\.Gptswarm: Language agents as optimizable graphs\.In*Forty\-first International Conference on Machine Learning*, 2024\.

## Appendix AAppendix

### A\.1Probe Set Construction for Capability Fingerprinting

##### Training\-query pool\.

Following prior work on automatic MAS design and orchestration\([Zhang et al\., 2025c](https://arxiv.org/html/2610.13144#bib.bib36);[Zhang et al\., 2025b](https://arxiv.org/html/2610.13144#bib.bib35)\), which usesB′∈\{40,80\}B^\{\\prime\}\\in\\\{40,80\\\}training queries per dataset, we adopt the 40\-query setting\. We randomly sample 40 queries from each of the six benchmark datasets, forming a pool of 240 training\-side queries\. Probe\-set construction and competence\-supervision data collection both draw from this pool, without introducing additional distinct task queries\. This matches the distinct\-query budget, not the offline LLM execution budget\.

##### Dataset\-specific splits\.

The dataset loader uses the MMLU development split for training queries and the validation split for evaluation queries\. For the other five benchmarks, including HumanEval, fixed dataset\-specific index files define the training and evaluation subsets\. The training and evaluation index sets in these files are disjoint, and the evaluation loader selects only the designated evaluation indices\.

##### Design goals\.

We construct the probe set to satisfy three requirements\. First, it should provide*clear difficulty resolution*, so that backbones of different competence levels can be meaningfully separated rather than collapsing to similar outcomes on the same probe set\. Second, it should provide*multi\-domain coverage*, so that the resulting capability fingerprint reflects multiple capability dimensions instead of only one narrow skill type\. Third, it should remain*compact*, so that one\-time fingerprint registration incurs only limited overhead\.

##### Anchor\-based difficulty calibration\.

To obtain a stable and interpretable difficulty scale, we evaluate four anchor backbones with progressively stronger standalone capability—Llama\-3\.2\-3B\-Instruct,Llama\-3\.1\-8B\-Instruct,GPT\-4o, andGPT\-5—on all 240 sampled queries\. For each itemxx, we record the binary pass vector

𝐬⁡\(x\)=\(s1,s2,s3,s4\),sk∈\{0,1\},\\mathbf\{s\}\(x\)=\(s\_\{1\},s\_\{2\},s\_\{3\},s\_\{4\}\),\\qquad s\_\{k\}\\in\\\{0,1\\\},\(19\)wheresk=1s\_\{k\}=1indicates that thekk\-th anchor solvesxxcorrectly, andsk=0s\_\{k\}=0otherwise\.

We order the four anchors from weaker to stronger and retain only items whose pass patterns are monotone under this ordering\. Intuitively, if a weaker anchor succeeds while a stronger anchor fails, the item is not suitable for reliable difficulty calibration and is therefore discarded as ambiguous\. We additionally remove all\-fail items\(0,0,0,0\)\(0,0,0,0\), since such items do not help locate the capability range of the anchors and may be confounded by ambiguity, annotation noise, or excessively difficult instances\.

The remaining monotone pass patterns define four relative difficulty levels, ranging from easiest to hardest: items solved by all four anchors\(1,1,1,1\)\(1,1,1,1\), items solved by all but the weakest anchor\(0,1,1,1\)\(0,1,1,1\), items solved only by the two strongest anchors\(0,0,1,1\)\(0,0,1,1\), and items solved only by the strongest anchor\(0,0,0,1\)\(0,0,0,1\)\. This yields an empirically grounded difficulty scale shared across candidate backbones\.

##### Domain stratification\.

To ensure that the probe set captures multiple capability dimensions, we further partition the candidate items into four coarse domains according to their source datasets:

Table 5:Domain assignment used in probe\-set construction\.
##### Balanced sampling and final probe set\.

Combining the four domains with the four difficulty levels yields 16 domain–difficulty cells\. We sample 3 items from each cell, resulting in a final probe set of 48 items\. For each domain, the selected probes are ordered from easy to hard as three items from each difficulty level, yielding 12 probes per domain in total\. This gives the4×124\\times 12layout used in capability fingerprinting: the four rows correspond to capability domains, and the 12 columns within each row form a shared easy\-to\-hard scale\.

Overall, this construction provides a compact but structured probe set that simultaneously offers clear difficulty resolution, multi\-domain coverage, and low one\-time registration cost\.

Table 6:Dataset descriptions and statistics\.

### A\.2Proxy Supervision Data Construction for Task\-Conditioned Competence Modeling

The goal of task\-conditioned competence modeling is to make the fused representation capture how competent a given LLM backbone is expected to be on the current query\. Since such query\-level competence is not directly annotated, we construct a proxy supervision signal from empirical task performance\. The intuition is straightforward: if a backbone succeeds more frequently on a query across repeated trials, it should be regarded as more competent on that query; conversely, a lower success rate indicates lower competence\.

We collect this supervision data offline from the same pool of 240 training\-side samples used in Appendix[A\.1](https://arxiv.org/html/2610.13144#A1.SS1)\. After excluding the 48 samples selected as the probe set, we use the remaining 192 samples to build the proxy supervision data\. Thus, the probe set is used only for Capability Fingerprint registration, while the non\-probe samples are used to estimate query\-level competence targets\.

For each backbonemmand each non\-probe queryqq, we repeatedly run the backboneR=10R=10times with decoding temperature set to0\.20\.2, and define the supervision target as the empirical success rate:

ym,q=1R​∑r=1Rzm,q\(r\),y\_\{m,q\}=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}z\_\{m,q\}^\{\(r\)\},\(20\)wherezm,q\(r\)∈\{0,1\}z\_\{m,q\}^\{\(r\)\}\\in\\\{0,1\\\}denotes the outcome of therr\-th trial\. We use a small non\-zero temperature to preserve limited stochastic variation across repeated runs while avoiding excessive randomness in the estimated competence targets\. For HumanEval,zm,q\(r\)z\_\{m,q\}^\{\(r\)\}is the pass/fail result under the benchmark unit tests, soym,qy\_\{m,q\}is an empirical estimate ofpass@1\. For all other benchmarks,zm,q\(r\)z\_\{m,q\}^\{\(r\)\}is the standard correctness indicator, andym,qy\_\{m,q\}therefore corresponds to the empirical accuracy on queryqq\.

To ensure that the supervision data covers diverse competence patterns, we construct it using 14 LLM backbones with varying scales, capability levels, and specialization characteristics, including general\-purpose, code\-specialized, math\-specialized, and reasoning\-oriented models \(Table[7](https://arxiv.org/html/2610.13144#A1.T7)\)\. GPT\-4o serves as a probe\-construction anchor \(Appendix[A\.1](https://arxiv.org/html/2610.13144#A1.SS1)\), but is not among these supervision backbones\.

For each backbone, we first obtain its Capability Fingerprint from the 48\-sample probe set\. We then pair this fingerprint with each non\-probe query and its corresponding targetym,qy\_\{m,q\}, yielding supervised instances of the form

\(Fingerprint⁡\(m\),q,ym,q\)\.\\big\(\\mathrm\{Fingerprint\}\(m\),\\ q,\\ y\_\{m,q\}\\big\)\.\(21\)To avoid an overly skewed target distribution and improve the relative coverage of different competence levels, we perform distribution\-aware sampling over these candidate instances with respect toym,qy\_\{m,q\}\. The resulting training set for task\-conditioned competence modeling contains 1060 instances\.

These proxy supervision data are used to train task\-conditioned competence modeling \(§[3\.2\.2](https://arxiv.org/html/2610.13144#S3.SS2.SSS2)\) and provide the supervision basis for downstream role activation \(§[3\.3\.1](https://arxiv.org/html/2610.13144#S3.SS3.SSS1)\)\. By supervising the fused representation against empirical query\-level success outcomes, the model is encouraged to encode competence\-relevant information that can directly support role selection and communication graph construction\.

Table 7:LLM backbones used for proxy\-supervision data construction\.##### Evaluation of the auxiliary prediction head\.

We evaluate the auxiliary success\-rate prediction head used to supervise task\-conditioned competence representation learning\. We use query\-disjoint training and evaluation partitions of the existing success\-rate supervision data\. All backbone–query pairs sharing the same query are assigned to a single partition, so no evaluation query appears in training or validation\. We train the Stage\-I module with BCE supervision and select checkpoints using validation MAE\. The evaluation covers 153 backbone–query pairs spanning 28 held\-out queries from backbones and domains represented during training\. Across three training seeds, the predicted success rates achieve a pooled Spearman correlation of0\.629±0\.0220\.629\\pm 0\.022with empirical success rates and an MAE of0\.234±0\.0100\.234\\pm 0\.010on the\[0,1\]\[0,1\]scale \(mean±\\pmstandard deviation across training seeds\)\. The prediction task provides auxiliary supervision for representation learning, while downstream orchestration is conditioned on the learned representation rather than the scalar success\-probability estimate\.

### A\.3Amortized Cost of Capability Fingerprinting

##### One\-time registration cost\.

Our efficiency comparison measures deployment\-time LLM token usage separately from offline supervision collection and model training\. The trained orchestration model is reused without further parameter updates\. A new backbone requires a one\-time fingerprint registration: evaluating it on the fixed 48\-item probe set to obtainVmV\_\{m\}, without repeating supervision collection or model training\. We measure this registration cost for the three performance\-oriented backbones in Section[4\.4](https://arxiv.org/html/2610.13144#S4.SS4):Gemini 3\.1 Pro,GLM 5\.1, andGPT\-4o\(Table[8](https://arxiv.org/html/2610.13144#A1.T8)\)\. The amortization below concerns this registration cost, not offline training\.

##### Average cost recovery\.

In Section[4\.4](https://arxiv.org/html/2610.13144#S4.SS4), the average online token usage is 3953\.94 tokens/query for G\-Designer and 2520\.26 tokens/query for our method, yielding an average saving of

Δsave=3953\.94−2520\.26=1433\.68\\Delta\_\{\\text\{save\}\}=3953\.94\-2520\.26=1433\.68\(22\)tokens per query\. Since this saving is averaged over the three performance\-oriented backbones, we use it only to compute an average amortization point\. Specifically, the average one\-time fingerprinting cost across the three backbones is 39,840 tokens, which corresponds to 830\.0 tokens per probe over the 48\-item probe set\. Therefore, the average cost\-recovery point is

Nbreak\-even=398401433\.68=27\.79\.N\_\{\\text\{break\-even\}\}=\\frac\{39840\}\{1433\.68\}=27\.79\.\(23\)That is, the one\-time cost of obtainingVmV\_\{m\}is recovered after only 27\.79 queries on average\. These results suggest that the probe set remains compact in practice and that the additional registration cost of capability fingerprinting can be amortized quickly in realistic multi\-query deployment settings\.

Table 8:One\-time token cost of Capability Fingerprinting for the three performance\-oriented backbones in Section[4\.4](https://arxiv.org/html/2610.13144#S4.SS4)\.

### A\.4Generalization to Harder Unseen Benchmarks with Gemma4

We further evaluate our trained model on two unseen datasets, GPQA Diamond\([Rein et al\., 2023](https://arxiv.org/html/2610.13144#bib.bib20)\)and BBEH\-mini\([Kazemi et al\., 2025](https://arxiv.org/html/2610.13144#bib.bib10)\), with an unseen LLM backbone, Gemma4\. Table[A\.4](https://arxiv.org/html/2610.13144#A1.SS4)reports task performance and collaboration cost\.

Table 9: Results on harder unseen benchmarks with Gemma4\. Accuracy is reported in percent\. Green annotations denote token savings relative to G\-Designer for the same benchmark and backbone\.

Results\.In the comparisons reported in Table[A\.4](https://arxiv.org/html/2610.13144#A1.SS4), Ours improves accuracy over G\-Designer on Gemma4 by 3\.03 points on GPQA and 2\.18 points on BBEH\-mini, while reducing tokens by 33\.6% on both benchmarks\. These results support the transfer of competence\-conditioned orchestration to these two unseen benchmarks with Gemma4 without additional training data\.

### A\.5Code Repository