OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

# OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems
Source: [https://arxiv.org/html/2609.21527](https://arxiv.org/html/2609.21527)
Xunkai Li\*Kaixiang ZhangAffiliation:Minghao An, Zekai Chen, Yuxuan Ba, Rong\-Hua LiAffiliation:\*Equal contribution\.

###### Abstract

Graph\-enhanced multi\-agent systems \(G\-MAS\) coordinate large language model agents through communication graphs and role assignments, which determine how agents exchange information and divide responsibilities\. However, final\-score comparisons across systems combine differences in models, communication patterns, roles, and computation costs, making performance differences difficult to attribute to specific communication structures, role assignments, and information flows\. To address this evaluation attribution problem, we introduce OpenMAS\-GCom, a benchmark for diagnosing how these components affect G\-MAS performance through controlled interventions\. We represent systems through collaboration units, communication links, shared intermediate information, and execution rules\. OpenMAS\-GCom compares original systems with versions modified by changing one component while keeping tasks, models, prompts, and budget limits fixed\. We rewire communication edges, remove specialist or critic agents, replace intermediate messages with incorrect content, and disable workers during execution\. The benchmark evaluates 17 single\-agent, ordinary multi\-agent, and graph\-enhanced configurations on 29 datasets across six domains\. We add 400 G\-MAS\-Complex tasks requiring agents to combine information from multiple documents, resolve conflicting records, and return specified values with source identifiers\. Experiments show larger mean losses after specialist removal than after critic removal, different performance degradation under incorrect messages and worker failures despite similar original scores, and different configurations achieving the highest accuracy and accuracy per token on G\-MAS\-Complex\.

## 1Introduction

Large language model \(LLM\) agents collaborate by assigning roles and exchanging intermediate results to solve mathematical reasoning, code generation, and question answering tasks\([Wu et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib5),[Du et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib6)\)\. In graph\-enhanced multi\-agent systems \(G\-MAS\), communication graphs determine which agents exchange information, while roles determine which agents generate, verify, or integrate solutions\([Zhuge et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib14),[Zhang et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib12)\)\. For example, one agent proposes an answer, another checks it, and a third combines intermediate results\. Performance therefore depends on both the underlying language model and how agents communicate and divide responsibilities\.

Existing evaluations mainly compare final task scores across systems\. Recent studies analyze information propagation across communication topologies, while MultiAgentBench evaluates task completion under different multi\-agent settings\([Shen et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib15),[Zhu et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib13)\)\. A higher score may accompany a stronger model, more agents, different communication links or roles, and greater computation costs\. These factors often change together across methods, making performance differences difficult to attribute to specific organizational choices\. This creates an evaluation attribution problem for G\-MAS\. How do communication structures, role composition, and information flow affect performance when other evaluation settings are held fixed?

To address this problem, we present OpenMAS\-GCom, a diagnostic benchmark using controlled organizational interventions\. We represent each system through collaboration units, communication links, shared intermediate information, and execution rules\. We compare original and modified systems after changing one component, with task inputs, model settings, prompts, and computation budget limits fixed\. We evaluate 17 single\-agent, ordinary multi\-agent, and graph\-enhanced configurations on 29 datasets across six domains through a common interface that records outputs, intermediate messages, executed units, and resource use\. We introduce 400 G\-MAS\-Complex tasks to examine how agents combine information from multiple documents, resolve conflicting records, and return specified values with source identifiers\. Four intervention protocols rewire communication edges while preserving node degrees, remove specialist or critic agents, replace intermediate messages with incorrect content, and disable workers during execution\.

Our Contributions\.\(1\)Comprehensive Benchmark\.We integrate datasets, baseline configurations, and four intervention protocols for evaluating changes to communication, roles, intermediate information, and worker availability under shared settings\. \(2\)Valuable Insights\.Across five datasets, specialist removal reduces the mean score by 3\.58 percentage points, compared with 0\.58 for critic removal\. MAD and VeriMap start with similar accuracy but respond differently to incorrect messages\. On G\-MAS\-Complex, different configurations achieve the highest accuracy and accuracy per token\. \(3\)Open\-sourced Benchmark Library\.We organize task loading, method execution, scoring, and interventions into reusable modules\. Run identifiers link configurations, predictions, intervention settings, and costs to support reproducible evaluation\.

## 2Problem Statement

OpenMAS\-GCom connects task evaluation with organizational diagnosis through two complementary pipelines\. The collaboration pipeline produces predictions and execution records through a common interface\. The diagnostic pipeline measures how these outcomes change when selected organizational components are modified\.

### 2\.1End\-to\-End Collaboration Pipeline

Given a task domainTTwith evaluation setDT=\{\(xi,yi\)\}i=1ND\_\{T\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, a methodFFspecifies organization construction and execution\. Under configurationλ\\lambda, its construction procedure produces𝒪i=Fbuild​\(xi,λ\)\\mathcal\{O\}\_\{i\}=F\_\{\\mathrm\{build\}\}\(x\_\{i\},\\lambda\)\. A fixed design uses the same organization across inputs, while task\-dependent construction can change its units and connections\. Each organization has the form

𝒪=\(V,E,S,π\),\\mathcal\{O\}=\(V,E,S,\\pi\),\(1\)whereVVspecifies runnable collaboration units,EEspecifies permitted directed communication,SSholds shared information and intermediate states, andπ\\pigoverns activation, routing, state updates, and termination\. A unit declares its role, local state, and input and output schemas\. Agent units form a subsetVA⊆VV\_\{A\}\\subseteq V, alongside tool or control units\. These components separate participation, communication, information access, and execution decisions\.

For inputxix\_\{i\}, sequential workflows activate units in order, parallel workflows activate several eligible units, and debate repeats message exchange through the policy\. Graph\-based execution uses the configured relations to select permitted transfers and activations\. The runner records the trace

τi=\(\(,,,,,\)\)t=1Ti,\\tau\_\{i\}=\(\(v\_\{t\},m\_\{t\},a\_\{t\}\)\)\_\{t=1\}^\{T\_\{i\}\},\(2\)wherevtv\_\{t\}is the executed unit,mtm\_\{t\}is its available message state, andata\_\{t\}is its output or control action\. The final outputoio\_\{i\}is extracted from the completed trace\. Associated records contain token usage, model calls, latency, executed units and relations, retries, and termination status\.

The representation captures the organizational properties exposed to the runner\. Method\-specific prompts, learned parameters, tool definitions, and controller logic remain in the configuration and policy\. Shared\-state permissions specify the information each unit may read or write\. The configured organization describes available units and relations, while the trace identifies those used for an input\. This distinction connects the comparison interface with configuration\-specific execution details\.

The task score is

s⁡\(F,𝒪T,T\)=1N​∑i=1NMT​\(P⁡\(oi\),yi\),s\(F,\\mathcal\{O\}\_\{T\},T\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}M\_\{T\}\(P\(o\_\{i\}\),y\_\{i\}\),\(3\)wherePPparses the output,MTM\_\{T\}is the task\-specific metric, and𝒪T=\{𝒪i\}i=1N\\mathcal\{O\}\_\{T\}=\\\{\\mathcal\{O\}\_\{i\}\\\}\_\{i=1\}^\{N\}collects the task\-specific organizations\. The parser and metric connect execution with comparable task scores\.

### 2\.2Two\-Stage Diagnostic Pipeline

The reference stage executes the original organization and records predictions, traces, and costs\. The intervention stage modifies one organizational component and repeats the evaluation on the same tasks\. Shared controls are

𝒞=\{DT,Θ,MT,P\},\\mathcal\{C\}=\\\{D\_\{T\},\\Theta,M\_\{T\},P\\\},\(4\)whereΘ\\Thetacontains the backend, prompts, decoding and routing settings, stopping rules, and task\-level execution and communication budget limits retained within each pair\. For intervention typerr, the modified organization is

𝒪r,i′=Ir​\(𝒪i,Δ​𝒪r,i\)\.\\mathcal\{O\}^\{\\prime\}\_\{r,i\}=I\_\{r\}\(\\mathcal\{O\}\_\{i\},\\Delta\\mathcal\{O\}\_\{r,i\}\)\.\(5\)The paired intervention response compares the resulting task scores

Δr​\(F,T\)=s⁡\(F,𝒪r,T′,T\)−s⁡\(F,𝒪T,T\)\.\\Delta\_\{r\}\(F,T\)=s\(F,\\mathcal\{O\}^\{\\prime\}\_\{r,T\},T\)\-s\(F,\\mathcal\{O\}\_\{T\},T\)\.\(6\)Here𝒪r,T′=\{𝒪r,i′\}i=1N\\mathcal\{O\}^\{\\prime\}\_\{r,T\}=\\\{\\mathcal\{O\}^\{\\prime\}\_\{r,i\}\\\}\_\{i=1\}^\{N\}collects the modified organizations\. The paired score change measures the average response to interventionrrunder the controls in Equation[4](https://arxiv.org/html/2609.21527#S2.E4)\. A negative value indicates lower performance after modification\. Its unit of analysis is the complete modified execution, including subsequent message processing and actions\. Structure, node, information, and execution interventions modify relations, units, intermediate messages, and worker availability, respectively\.

## 3Benchmark Design

OpenMAS\-GCom integrates task selection, organization configuration, collaborative execution, and paired diagnosis\. Figure[1](https://arxiv.org/html/2609.21527#S3.F1)connects task inputs, execution records, and metrics\.

![Refer to caption](https://arxiv.org/html/2609.21527v1/opengmas_framework_ppt.png)Figure 1:OpenMAS\-GCom model framework\. The benchmark selects datasets, configures G\-MAS organizations, executes graph guided collaboration, and applies four graph intervention protocols\.### 3\.1Task and Dataset Ecosystem

Broad Task Suite\.The benchmark covers 29 datasets across mathematics, knowledge, code generation, biomedical question answering, finance, and tabular reasoning\. These domains combine different requirements for sequential reasoning, specialist knowledge, executable checking, and structured information integration\. Dataset loaders provide task inputs, reference answers, and metric metadata to the common runner\.

G\-MAS\-Complex Construction\.G\-MAS\-Complex contains 400 frozen multi\-document tasks in four complexity tiers\. Document dependencies distribute relevant information across sources, conflicting records require revision or authority resolution, and structured outputs require consistency checks\. These properties combine retrieval, reconciliation, reasoning, and verification within each task\. Comparisons measure how the evaluated configurations handle these coupled requirements\. Tier labels describe task construction, and fixed\-method scores measure empirical difficulty\.

The four tiers contain 100 tasks each and share a structured answer contract\. In an illustrative inventory task, D1 records 12 units, D2 revises the inventory to 9, and D3 reserves 4\. The answer requires 5 available units and source identifiers D2 and D3\. Selecting D1 gives 8 units, while omitting D2 produces an incomplete source record\. Strict exact match checks required fields, values, rankings, references, checkpoints, and checksum constraints together\. Appendix[B\.1](https://arxiv.org/html/2609.21527#A2.SS1)expands this example and distinguishes content errors from output\-format failures\.

### 3\.2Algorithm Taxonomy

Definition of G\-MAS\.A graph\-enhanced configuration contains at least two agent units and an explicit directed graph that constrains routing, activation, verification flow, or execution order\. Classification uses the graph exposed to the adapter and the runtime decision controlled by its edges\. Fixed, task\-generated, and learned graphs can satisfy this criterion\. Equation[1](https://arxiv.org/html/2609.21527#S2.E1)supplies the representation used to inspect these properties\.

Single\-Agent and Ordinary Multi\-Agent Configurations\.Single\-agent configurations use one agent without inter\-agent communication, represented here by DeepSeek and Qwen\([DeepSeek\-AI, 2025](https://arxiv.org/html/2609.21527#bib.bib43),[Qwen Team, 2025](https://arxiv.org/html/2609.21527#bib.bib44)\)\. Ordinary multi\-agent configurations coordinate agents through their dialogue or collaboration policies\. The tables place the evaluated AutoGen, MADebate, and DyLAN configurations in this group\([Wu et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib5),[Du et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib6),[Liu et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib31)\)\. These labels identify the reported configurations, while the adapter description specifies the organization and execution mechanism for each method\.

Graph\-Enhanced Configurations\.The selected graph\-enhanced configurations cover graph construction, sparse interaction, routing, verification, and learned coordination\. Table[5](https://arxiv.org/html/2609.21527#A1.T5)lists the corresponding methods and sources\. Configuration records describe graph materialization, construction rules, runtime use, role heterogeneity, and recovery\. These attributes connect method selection with the intervention protocols\.

GPTSwarm optimizes prompts and graph connectivity\([Zhuge et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib14)\), while G\-Designer learns a generator for task\-dependent communication graphs\([Zhang et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib12)\)\. Configuration attributes further distinguish topology construction, runtime updates, learned components, role heterogeneity, verification, and recovery\. Appendix[B\.4](https://arxiv.org/html/2609.21527#A2.SS4)defines these attributes and links graphs with execution\.

### 3\.3Graph Intervention Protocols

Each protocol modifies one organizational factor under the shared controls of Equation[4](https://arxiv.org/html/2609.21527#S2.E4)\. The four protocols examine where information travels, which units process it, what they receive, and how execution continues when workers become unavailable\.

Structure Intervention\.The structure protocol rewires directed edges while preserving the agent set, in\-degree and out\-degree sequences, edge count, and communication budget limit\. A directed edge swap exchanges the destinations of two connections, changing communication partners while retaining local degrees\. Self\-loops, duplicate edges, and invalid role connections are rejected\. The score change measures the response to this modification\. Executed calls and messages describe how the resulting graph uses its available budget\.

Node Intervention\.The node protocol removes selected collaboration units and their incident edges\. The No critic and No specialist variants target different role groups\. Expert only retains the expert nodes and their induced connections\. Each variant links its available functions and communication relations with its task score\.

Information Intervention\.The information protocol replaces eligible upstream records or intermediate candidates with format\-valid incorrect candidates\. The modified content reaches downstream units through the existing graph and routing rules\. The corruption rate is the fraction of eligible items replaced\. This protocol measures final\-answer accuracy under incorrect intermediate inputs\.

Execution Intervention\.The execution protocol disables workers during runtime and resumes from the last valid shared state through configured retry, fallback, or rerouting procedures\. Increasing the failure count tests how larger disruptions affect task completion\. Appendix[A\.2](https://arxiv.org/html/2609.21527#A1.SS2)provides the intervention settings and shared controls\.

### 3\.4Evaluation Protocols

Utility\.We compare single\-agent, ordinary multi\-agent, and graph\-enhanced configurations on shared tasks\. Single\-agent and ordinary multi\-agent results provide reference scores for assessing the utility of each configuration\. These system comparisons include the complete reasoning and execution procedure\. Paired interventions examine changes within a configured organization\.

Effectiveness\.We examine task scores and method rankings across the six domains and G\-MAS\-Complex\. Domain summaries describe performance across related tasks, while individual dataset scores capture more specific requirements\. These views identify the strongest configurations and the separation among competing methods\.

Robustness\.We compare each modified score with the same configuration’s reference score\. Absolute score change measures the difference in task units, and relative retention divides the modified score by a positive reference score\. Rewiring ratios, unit\-removal variants, corruption rates, and worker\-failure counts define the intervention settings\.

Efficiency\.We record token usage, model calls, latency, executed units and relations, and retries\. For G\-MAS\-Complex, strict exact match describes successful completion, while token usage and latency describe execution requirements\. All four dimensions use the common execution interface and task\-specific scoring rules\.

## 4Experiments and Analyses

Three questions organize the evaluation\.Task utilitycovers Q1 to Q3\.Organizational sensitivitycovers Q4 to Q6\.Failure robustness and resource usecovers Q7 and Q8\. The following experiments examine each group through specific comparisons\.Q1 \(Utility\)\.Which evaluated configurations improve task scores over single\-agent baselines?Q2 \(Utility\)\.How do the best evaluated graph\-enhanced configurations compare with ordinary multi\-agent systems?Q3 \(Effectiveness\)\.How do existing G\-MAS methods perform across different task domains?Q4 \(Structure Intervention\)\.Does topology matter when the degree sequence and communication budget limits are fixed?Q5 \(Unit Intervention\)\.How much do role and collaboration\-unit removals affect performance?Q6 \(Information Intervention\)\.How does accuracy change as the fraction of incorrect upstream items increases?Q7 \(Execution Intervention\)\.How robust are G\-MAS executions as the number of failed workers increases under recovery?Q8 \(Efficiency\)\.How efficient are G\-MAS methods?

### A\. Experimental Setup

Datasets\. We evaluate seven categories\. \(1\) Mathematics uses GSM8K and AQuA\([Cobbe et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib1),[Ling et al\., 2017](https://arxiv.org/html/2609.21527#bib.bib16)\)\. \(2\) Knowledge uses MMLU\-Pro and StrategyQA\([Wang et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib11),[Geva et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib17)\)\. \(3\) Code uses HumanEval and LiveCodeBench\([Chen et al\., 2021a](https://arxiv.org/html/2609.21527#bib.bib7),[Jain et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib8)\)\. \(4\) Biomedical QA uses MedQA and MedMCQA\([Jin et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib18),[Pal et al\., 2022](https://arxiv.org/html/2609.21527#bib.bib19)\)\. \(5\) Finance uses ConvFinQA and FinQA\([Chen et al\., 2022](https://arxiv.org/html/2609.21527#bib.bib20),[Chen et al\., 2021b](https://arxiv.org/html/2609.21527#bib.bib21)\)\. \(6\) Tabular reasoning uses TabFact and WikiTableQuestions\([Chen et al\., 2020](https://arxiv.org/html/2609.21527#bib.bib23),[Pasupat and Liang, 2015](https://arxiv.org/html/2609.21527#bib.bib22)\)\. \(7\) G\-MAS\-Complex contains 400 frozen multi\-document tasks in four tiers\. Full definitions are in Appendix[A\.1](https://arxiv.org/html/2609.21527#A1.SS1)\.

Baselines\. We evaluate the 17 configurations in Table[11](https://arxiv.org/html/2609.21527#A1.T11)through common adapters and evaluation controls\. Appendix[A\.2](https://arxiv.org/html/2609.21527#A1.SS2)describes execution, and Appendix[B\.5](https://arxiv.org/html/2609.21527#A2.SS5)defines validation records\.

Scores and variation\.The broad suite retains dataset\-specific metrics, and G\-MAS\-Complex uses Strict EM\. Table[1](https://arxiv.org/html/2609.21527#S4.T1)reports central scores\. Appendix[A\.1](https://arxiv.org/html/2609.21527#A1.SS1)separates sample\-set variation, run variation, recorded estimates, and binomial standard errors\. Rankings in Q1 to Q3 describe displayed central values\. Intervention analyses report paired score changes and reference\-score retention, while efficiency compares observed scores and resource use\.

### B\. Utility of Multi\-Agent Collaboration \(Q1\)

To answer Q1, we compare Single\-Agent, Ordinary MAS, and Graph\-enhanced MAS on 13 datasets\.

Table 1:Central\-score comparison \(%\)\. Bold and underlined entries mark the highest and second\-highest distinct displayed scores, including ties\. Appendix[A\.8](https://arxiv.org/html/2609.21527#A1.SS8)defines the statistical sources\.MathematicalReasoningKnowledge &CommonsenseCodeGenerationBiomedicalQAFinancialReasoningTabularReasoningComplexCollaborationMethodGSM8KAQuAMMLU\-ProStrategyQAHumanEvalLiveCodeMedQAMedMCQAConvFinQAFinQATabFactWTQG\-MAS\-ComplexDeepSeek93\.0393\.0389\.8989\.8973\.0773\.0784\.6084\.6083\.5483\.5421\.9021\.9088\.5888\.5881\.2781\.2760\.1360\.1338\.1038\.1091\.4791\.4776\.7376\.7326\.7526\.75Qwen59\.4459\.4468\.9068\.9060\.9360\.9379\.4079\.4090\.2490\.2426\.8626\.8678\.9578\.9573\.5373\.5361\.8161\.8128\.5128\.5181\.4081\.4065\.2765\.2713\.2513\.25AutoGen92\.9292\.9290\.1690\.1671\.8071\.8084\.1384\.1381\.7181\.7118\.8618\.8688\.7488\.7479\.6779\.6760\.6760\.6736\.7036\.7091\.6791\.6775\.9375\.9323\.5023\.50MADebate93\.2093\.2089\.8989\.8971\.8771\.8784\.2084\.2079\.8879\.8820\.3820\.3889\.4589\.4579\.6779\.6761\.0161\.0136\.9736\.9791\.4791\.4776\.6776\.6714\.0014\.00DyLAN93\.3393\.3389\.8989\.8973\.8773\.8786\.6786\.6785\.9885\.9822\.8622\.8688\.4588\.4579\.9379\.9361\.1061\.1038\.8838\.8892\.5392\.5377\.4077\.404\.504\.50GPTSwarm93\.6693\.6691\.3491\.3471\.2071\.2085\.6785\.6785\.5785\.5721\.1421\.1489\.6389\.6380\.2080\.2061\.1261\.1238\.6238\.6292\.6092\.6076\.6076\.6015\.5015\.50GDesigner93\.6693\.6689\.6389\.6371\.8771\.8785\.0085\.0085\.3785\.3723\.4323\.4388\.8588\.8579\.0079\.0061\.5461\.5438\.5138\.5191\.7391\.7380\.3380\.3314\.5014\.50SparseCT93\.8693\.8689\.7689\.7671\.3371\.3385\.8085\.8085\.9885\.9820\.7620\.7688\.9288\.9279\.6779\.6760\.8160\.8139\.2339\.2391\.9391\.9377\.4777\.4715\.2515\.25R\-GFM93\.6393\.6390\.0390\.0371\.5371\.5384\.4084\.4089\.0289\.0221\.5221\.5289\.4789\.4779\.3379\.3360\.6360\.6338\.7138\.7192\.1392\.1379\.8079\.802\.252\.25BigMAS93\.4593\.4589\.6389\.6372\.2072\.2085\.5385\.5385\.7785\.7721\.3321\.3389\.8789\.8778\.7378\.7360\.6960\.6938\.3638\.3692\.3392\.3376\.8076\.8019\.2519\.25GoAgent93\.2393\.2389\.7689\.7670\.3370\.3385\.2785\.2779\.8879\.8821\.1421\.1489\.4089\.4079\.1379\.1360\.4960\.4937\.2037\.2092\.3392\.3376\.2076\.2050\.2550\.25ARGDes93\.4893\.4889\.6389\.6371\.4771\.4786\.0086\.0089\.6389\.6321\.3321\.3388\.7788\.7779\.2079\.2060\.2060\.2038\.7138\.7191\.9391\.9377\.0077\.0013\.2513\.25EIB93\.3393\.3389\.3789\.3772\.4772\.4785\.4085\.4083\.5483\.5421\.3321\.3389\.0089\.0079\.0079\.0061\.3461\.3438\.5438\.5492\.3392\.3379\.3379\.3314\.0014\.00

In Table[1](https://arxiv.org/html/2609.21527#S4.T1), graph\-enhanced configurations have the highest displayed central score on 7 of 13 datasets, single\-agent configurations on 4, and ordinary multi\-agent configurations on 2\. Single\-agent models have the highest central scores on both displayed code datasets and ConvFinQA\. This distribution places configuration choice in the context of the target task\. Our Conclusion \(C1\)\. Central\-score advantages vary across tasks and configuration groups\.

### C\. Competitiveness of Graph\-Enhanced MAS \(Q2\)

For Q2, Figure[2](https://arxiv.org/html/2609.21527#S4.F2)displays group\-wise maxima over 3 ordinary and 12 graph\-enhanced configurations\. Each bar summarizes attainable reported scores in its group and task subset\. The selected configuration can change across domains and tiers, so the comparison describes the best displayed score in each setting\.

Figure 2:Group\-wise maximum reported scores over evaluated Ordinary MAS and Graph\-enhanced MAS configurations\. Each bar selects the highest score within its group and task subset, allowing different configurations to supply different bars\.On the four G\-MAS\-Complex tiers, the plotted margins are 15, 17, 31, and 45 percentage points\. Across all 400 tasks, GoAgent and VeriMap\([Chen et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib24),[Xu et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib39)\)score above the single\-agent DeepSeek reference, while the other ten graph\-enhanced configurations score below it\. Tier maxima describe the best attained scores within each subset\. Our Conclusion \(C2\)\. Strong complex\-task performance is concentrated in a small subset of the evaluated graph\-enhanced configurations\.

### D\. Performance Across Task Domains \(Q3\)

Figure[3](https://arxiv.org/html/2609.21527#S4.F3)compares four graph\-enhanced configurations across six domains\.

Code ranges from 53\.16% to 57\.06%, giving the largest central\-score spread of 3\.90 percentage points\. The spreads are 1\.25 on Tabular, 1\.06 on Mathematics, 0\.53 on Biomedical QA, 0\.52 on Finance, and 0\.51 on Knowledge\. Appendix[A\.1](https://arxiv.org/html/2609.21527#A1.SS1)reports dataset\-level variation\. Our Conclusion \(C3\)\. Displayed central\-score dispersion varies substantially across task domains\.

Figure 3:Domain summaries with independent panel scales\. Accuracy denotes the reported aggregate score\. Appendix[A\.1](https://arxiv.org/html/2609.21527#A1.SS1)lists dataset\-specific metrics\.Figure 4:Responses to topology rewiring, role removal, incorrect messages, and worker failures\. Panels \(a\) and \(b\) show five datasets, while panels \(c\) and \(d\) compare the displayed methods\. Section[3\.3](https://arxiv.org/html/2609.21527#S3.SS3)defines the corresponding operations\.
### E\. Sensitivity to Communication Structure \(Q4\)

We hold agents, roles, degrees, edge count, models, prompts, routing policy, and budget limits fixed\. Rewiring ratios are 10%, 30%, 50%, and 70%\. The five\-dataset mean falls from 81\.88% to 81\.21%, 79\.54%, 77\.21%, and 74\.99%\. At 70% rewiring, the loss is 6\.89 percentage points, leaving 91\.59% of the reference score\. The progressive decrease associates task performance with the arrangement of communication partners in the evaluated setting\. Actual calls, message counts, and context lengths characterize execution after rewiring\. Our Conclusion \(C4\)\. Degree\-preserving rewiring reduces scores under the specified protocol\.

### F\. Sensitivity to Collaboration\-Unit Removal \(Q5\)

To answer Q5, we compare Base with No critic, No specialist, and Expert only\. The first two remove the selected role nodes and incident edges\. Expert only keeps the expert\-induced subgraph\. The mean falls from 81\.88% to 81\.30%, 78\.30%, and 75\.59%, respectively, giving losses of 0\.58, 3\.58, and 6\.29 percentage points\. Specialist removal incurs 6\.17 times the mean loss of critic removal, while MMLU\-Pro improves after critic removal\. Our Conclusion \(C5\)\. Removing different collaboration units produces asymmetric score changes, with the largest loss under Expert only\.

### G\. Robustness to Incorrect Messages \(Q6\)

We replace eligible upstream records or candidates with format\-valid incorrect candidates at corruption rates of 0%, 20%, 40%, 60%, and 80%\. The graph and execution controls remain fixed\. The concentration axis in Figure[4](https://arxiv.org/html/2609.21527#S4.F4)\(c\) denotes this item replacement rate\. MAD falls from 85\.30% to 44\.41%, and VeriMap falls from 85\.50% to 54\.59%\. At 80%, the retained scores are 52\.06% and 63\.85%, respectively\. Their accuracy gap grows from 0\.20 to 10\.18 percentage points\. Our Conclusion \(C6\)\. Similar reference scores conceal different corruption responses\.

Figure 5:G\-MAS\-Complex efficiency\. Bars show strict exact match per 1K tokens for all baselines\.
### H\. Recovery Under Worker Failures \(Q7\)

To answer Q7, we disable 0 to 4 workers and continue through each method’s configured retry, fallback, or rerouting procedure\. GoAgent decreases from 84\.22% to 76\.80%, and AutoGen decreases from 83\.82% to 73\.80%\. With four failed workers, the losses are 7\.42 and 10\.02 percentage points, respectively\. The gap grows from 0\.40 to 3\.00 points across the sweep\. This profile compares task completion under the tested failure settings and continuation rules\. Our Conclusion \(C7\)\. GoAgent retains more reference accuracy than AutoGen in this worker\-failure setting\.

### I\. Efficiency \(Q8\)

For Q8, we compare observed operating points on G\-MAS\-Complex\. Table[2](https://arxiv.org/html/2609.21527#S4.T2)places Strict EM alongside tokens, latency, and executed nodes and relations\. Strict EM/K Tokens divides the strict\-match percentage by mean tokens per task in thousands\.

Table 2:Observed operating points on G\-MAS\-Complex\. Strict EM is in percent, tokens per task in K, and latency in seconds\. Nodes and edges are execution averages\. Strict EM/K Tokens describes score per 1K tokens at the reported setting\.MethodStrict EM\(%\)Tokens/task\(K\)Strict EM/KTokensLatency\(s\)NodesEdgesGoAgent50\.2526\.471\.9045\.4617\.2567\.15VeriMap29\.504\.736\.247\.094\.004\.00DeepSeek26\.755\.944\.501\.511\.000\.00AutoGen23\.503\.396\.936\.423\.002\.00BigMAS19\.2511\.651\.6517\.1114\.3030\.90GPTSwarm15\.503\.844\.046\.483\.952\.95SparseCT15\.254\.303\.559\.254\.003\.00GDesigner14\.506\.352\.287\.956\.159\.05EIB14\.005\.202\.699\.256\.158\.05GraphSearch14\.008\.061\.7411\.849\.1518\.40MADebate14\.003\.803\.685\.854\.003\.00ARGDes13\.257\.051\.888\.757\.1511\.25Qwen13\.256\.352\.096\.411\.000\.00GraphR110\.7510\.351\.0412\.6011\.1528\.55MASRouter10\.7513\.210\.8110\.7621\.6038\.20DyLAN4\.507\.220\.6212\.679\.1512\.10R\-GFM2\.259\.060\.2521\.2710\.1513\.10

Figure[5](https://arxiv.org/html/2609.21527#S4.F5)ranks observed score\-per\-token values\. AutoGen and VeriMap have the highest ratios, while GoAgent has the highest Strict EM\. Compared with DeepSeek, GoAgent gains 23\.50 percentage points while using 4\.46 times the tokens and 30\.11 times the latency\. Tokens describe model usage, and latency describes elapsed execution under the associated deployment settings\. Our Conclusion \(C8\)\. The highest accuracy and highest score per token occur at different operating points\.

## 5Conclusion and Future Directions

OpenMAS\-GCom evaluates how graph\-enhanced multi\-agent systems respond to organizational changes\. Its executable representation separates collaboration units, communication relations, shared information, and execution policy\. Four paired intervention families modify structure, unit composition, intermediate messages, and worker availability under shared controls\. The broad task suite and G\-MAS\-Complex connect these responses with task outcomes and resource use\. The shared interface associates each comparison with its organization settings, intervention parameters, task scores, and execution records\.

Three findings organize the evaluated results\. First, central\-score advantages vary across tasks, and GoAgent and VeriMap are the two graph\-enhanced configurations above DeepSeek on G\-MAS\-Complex\. Second, degree\-preserving rewiring and collaboration\-unit removal produce unequal score changes under their respective settings\. At 70% rewiring, the mean score declines by 6\.89 points, while specialist removal produces a larger mean decline than critic removal\. Third, similar reference scores accompany different responses to incorrect messages and unavailable workers\. MAD and VeriMap separate by 10\.18 points at 80% corruption, and the accuracy gap between GoAgent and AutoGen increases with worker failures\. The configuration with the highest Strict EM also differs from the configuration with the highest score per token\.

These findings characterize the tested configurations, tasks, and disruption settings\. Paired score changes summarize the response of the complete modified execution\. G\-MAS\-Complex targets document dependencies, record conflicts, and structured completion, while the broad suite retains each dataset’s task\-specific metric\. Resource comparisons describe observed execution settings through scores, token usage, and latency\. These views guide configuration selection and organizational diagnosis\. Extensions to longer\-horizon, interactive, and multimodal tasks can use the same organization interface and paired comparison protocol\.

## Appendix AAppendix

This appendix describes datasets, baseline adapters, metrics, interventions, evaluation strategies, execution metadata, parameter settings, supplementary results, and reproducibility records\.

### A\.1Dataset Description

OpenMAS\-GCom contains a broad task suite and a separately constructed G\-MAS\-Complex stress suite\. The broad suite contains 29 datasets across six domains\. Mathematics evaluates arithmetic, symbolic, and competition\-style reasoning with GSM8K, SVAMP, MultiArith, AQuA, ASDiv\-A, Game\-of\-24, and MATH\([Cobbe et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib1),[Patel et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib2),[Roy and Roth, 2015](https://arxiv.org/html/2609.21527#bib.bib37),[Ling et al\., 2017](https://arxiv.org/html/2609.21527#bib.bib16),[Miao et al\., 2020](https://arxiv.org/html/2609.21527#bib.bib36),[Yao et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib4),[Hendrycks et al\., 2021b](https://arxiv.org/html/2609.21527#bib.bib3)\)\. Knowledge and commonsense uses MMLU, MMLU\-Redux, MMLU\-Pro, StrategyQA, CommonsenseQA, ARC\-Easy, and ARC\-Challenge to test subject knowledge, robust multiple\-choice answering, implicit reasoning, and science QA\([Hendrycks et al\., 2021a](https://arxiv.org/html/2609.21527#bib.bib28),[Gema et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib42),[Wang et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib11),[Geva et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib17),[Talmor et al\., 2019](https://arxiv.org/html/2609.21527#bib.bib38),[Clark et al\., 2018](https://arxiv.org/html/2609.21527#bib.bib25)\)\. Code uses HumanEval, HumanEval\+\+, LiveCodeBench\-v6, and MultiAgentBench\-Coding for executable program synthesis and collaborative coding evaluation\([Chen et al\., 2021a](https://arxiv.org/html/2609.21527#bib.bib7),[Liu et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib26),[Jain et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib8),[Zhu et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib13)\)\. Biomedical tasks include PubMedQA, MedQA, MedMCQA, and the medical subset of MMLU\([Jin et al\., 2019](https://arxiv.org/html/2609.21527#bib.bib29),[Jin et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib18),[Pal et al\., 2022](https://arxiv.org/html/2609.21527#bib.bib19),[Hendrycks et al\., 2021a](https://arxiv.org/html/2609.21527#bib.bib28)\)\. Finance uses TAT\-QA, ConvFinQA, and FinQA to evaluate numerical reasoning over financial text and tables\([Zhu et al\., 2021](https://arxiv.org/html/2609.21527#bib.bib9),[Chen et al\., 2022](https://arxiv.org/html/2609.21527#bib.bib20),[Chen et al\., 2021b](https://arxiv.org/html/2609.21527#bib.bib21)\)\. Tabular reasoning uses TabFact, WikiSQL, WikiTableQuestions, and SQA for table verification, semantic parsing, and sequential table QA\([Chen et al\., 2020](https://arxiv.org/html/2609.21527#bib.bib23),[Zhong et al\., 2017](https://arxiv.org/html/2609.21527#bib.bib10),[Pasupat and Liang, 2015](https://arxiv.org/html/2609.21527#bib.bib22),[Iyyer et al\., 2017](https://arxiv.org/html/2609.21527#bib.bib41)\)\.

Table 3:Broad\-suite dataset summary\.Domain\-level statistics summarize the broad suite\. Per\-dataset sample counts are shown in Table[11](https://arxiv.org/html/2609.21527#A1.T11)\.Domain\#DatasetsDatasetsPrimary evaluation targetMathematics7GSM8K, SVAMP, MultiArith, AQuA, ASDiv\-A, Game\-of\-24, MATHNumeric answer, multiple\-choice answer, valid expression, or competition\-style final answerKnowledge and commonsense7MMLU, MMLU\-Redux, MMLU\-Pro, StrategyQA, CommonsenseQA, ARC\-Easy, ARC\-ChallengeMultiple\-choice or yes/no answer after robust answer extractionCode4HumanEval, HumanEval\+\+, LiveCodeBench\-v6, MultiAgentBench\-CodingUnit\-test pass rate, executable correctness, or code\-task quality scoreBiomedical4PubMedQA, MedQA, MedMCQA, MMLU\-MedBiomedical yes/no/maybe answer or medical multiple\-choice answerFinance3TAT\-QA, ConvFinQA, FinQANormalized numerical or textual financial answerTabular4TabFact, WikiSQL, WikiTableQuestions, SQATable entailment, table\-query answer, or sequential table answerG\-MAS\-Complex\.We additionally construct 400 frozen multi\-document tasks to stress collaboration rather than isolated recall\. Each task contains a query, source documents, dependency metadata, conflict annotations, and a machine\-checkable JSON output contract\. The documents have overlapping contents and distinct functions\. Some provide authoritative revisions, some provide intermediate records, some are distractors, and some define constraints that must be combined before the final answer can be produced\. This design makes the suite sensitive to delegation, provenance tracking, conflict resolution, and final verification\.

Table 4:G\-MAS\-Complex tier construction\.Each tier contains 100 frozen tasks\.TierMain stress factorDocument and conflict patternRequired behaviorMediumMulti\-document lookupSeveral documents with local dependencies and light distractorsRetrieve relevant records and return a structured answerHardCross\-document dependencyChained records with competing revisions or aliasesResolve dependencies before producing fields and referencesUltraConflict and provenanceMultiple plausible records, authority rules, and source checkpointsSelect authoritative records and justify structured fieldsExtremeEnd\-to\-end auditDense dependencies, distractors, revision conflicts, and checksum constraintsCombine retrieval, conflict resolution, ranking, and verification

For each task, the target answer is represented as

zi=\(fi,vi,ri,ℓi,hi,χi\),z\_\{i\}=\(f\_\{i\},v\_\{i\},r\_\{i\},\\ell\_\{i\},h\_\{i\},\\chi\_\{i\}\),\(7\)wherefif\_\{i\}contains required fields,viv\_\{i\}contains normalized values,rir\_\{i\}contains ranking or ordering requirements,ℓi\\ell\_\{i\}contains source references,hih\_\{i\}contains checkpoint records, andχi\\chi\_\{i\}contains checksum\-style consistency constraints\. The output parser first checks whether the response can be mapped to the requested JSON schema, then normalizes scalar values, lists, and source identifiers\. A task is counted as correct only when every required component satisfies the contract\.

Complex\-task validation example\.Each stress\-suite instance can also be written asxi=\(qi,𝒟i,Γi,Ωi\)x\_\{i\}=\(q\_\{i\},\\mathcal\{D\}\_\{i\},\\Gamma\_\{i\},\\Omega\_\{i\}\), whereqiq\_\{i\}is the task query,𝒟i\\mathcal\{D\}\_\{i\}is the source\-document collection,Γi\\Gamma\_\{i\}specifies dependencies and record conflicts, andΩi\\Omega\_\{i\}defines the output contract\. For example, an inventory task may contain three records\.D1D\_\{1\}states that 12 units are available,D2D\_\{2\}revises the inventory to 9 units, andD3D\_\{3\}reserves 4 units\. The correct answer is therefore 5 available units with source identifiersD2D\_\{2\}andD3D\_\{3\}\. UsingD1D\_\{1\}instead ofD2D\_\{2\}yields a content error, while returning the right value without the required source identifiers yields a contract error\. The strict scorer rejects both cases because it checks field values, provenance, and schema validity jointly\.

### A\.2Baseline Description

Every method is wrapped as an executable organization𝒪=\(V,E,S,π\)\\mathcal\{O\}=\(V,E,S,\\pi\)before evaluation\. This adapter layer separates method\-specific collaboration behavior from shared benchmark components such as dataset loading, backend calls, answer parsing, metric calculation, trace logging, and cost accounting\. Single\-agent baselines contain one runnable unit and no communication edges\. Ordinary MAS baselines contain multiple units with predefined collaboration patterns\. Graph\-enhanced MAS baselines expose organization graphs, learned or searched routing, sparse communication, verification links, or recovery policies through the same runner interface\.

Table 5:Baseline families and organization adapters\.FamilyMethodsAdapter representationSingle\-agentDeepSeek, Qwen\([DeepSeek\-AI, 2025](https://arxiv.org/html/2609.21527#bib.bib43),[Qwen Team, 2025](https://arxiv.org/html/2609.21527#bib.bib44)\)One runnable unit, empty edge set, direct answer parserOrdinary MASAutoGen, MADebate, DyLAN\([Wu et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib5),[Du et al\., 2023](https://arxiv.org/html/2609.21527#bib.bib6),[Liu et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib31)\)Fixed role set with predefined debate, refinement, or dynamic selection policyGraph construction and routingGPTSwarm, GDesigner, GraphSearch, GraphR1, MASRouter\([Zhuge et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib14),[Zhang et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib12),[Liu et al\., 2026b](https://arxiv.org/html/2609.21527#bib.bib32),[Luo et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib34),[Yue et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib35)\)Explicit graph or search/routing module mapped to typed directed relationsSparse or role\-specialized G\-MASSparseCT, R\-GFM, BigMAS, ARGDes, EIB\([Li et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib40),[Liu et al\., 2026a](https://arxiv.org/html/2609.21527#bib.bib33),[Hao et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib27),[Li et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib30),[Shen et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib15)\)Specialist units and sparse communication structures mapped to runnable nodes and edgesVerification and recovery\-oriented G\-MASGoAgent, VeriMap\([Chen et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib24),[Xu et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib39)\)Verification, fallback, or rerouting policies exposed as execution\-policy componentsOrganization configuration and execution\.The configured organization and the realized execution trace are recorded separately\. The organization specifies available units, roles, communication relations, routing policy, and recovery policy, while the trace records which units actually execute for a particular input\. Execution can differ from the static configuration as messages arrive, candidate answers are rejected, workers become unavailable, or recovery procedures are triggered\. This distinction separates available units from executed units and connects the graph specified by a method with the communication actually used to produce a prediction\.

### A\.3Metric Description

OpenMAS\-GCom reports the official or commonly adopted metric for each dataset whenever possible\. For classification and question\-answering datasets, the parser maps free\-form generations to the answer space before applying exact agreement with the reference\. For code tasks, generated programs are evaluated with the corresponding pass@1\-style executable test protocol\. For tabular and financial reasoning, answers are normalized before comparison, including number formatting, option markers, and short textual aliases when supported by the original benchmark metric\.

For a dataset𝒟=\{\(xi,yi\)\}i=1n\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}and methodFF, the broad\-suite score is

s⁡\(F,𝒟\)=1n​∑i=1nm⁡\(parse⁡\(F⁡\(xi\)\),yi\),s\(F,\\mathcal\{D\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}m\(\\operatorname\{parse\}\(F\(x\_\{i\}\)\),y\_\{i\}\),\(8\)wheremmis the dataset\-specific metric\. For QA datasets,mmis accuracy or exact match\. For code datasets,mmis executable correctness\. For MultiAgentBench\-Coding,mmis CodeQ\.

For G\-MAS\-Complex, strict exact match is

StrictEM\(z^i,zi\)=𝟏\[z^i⊧Ωi∧z^i=zi\],\\operatorname\{StrictEM\}\(\\hat\{z\}\_\{i\},z\_\{i\}\)=\\mathbf\{1\}\[\\hat\{z\}\_\{i\}\\models\\Omega\_\{i\}\\land\\hat\{z\}\_\{i\}=z\_\{i\}\],\(9\)and the suite score is

scomplex​\(F\)=1400​∑i=1400StrictEM⁡\(z^i,zi\)\.s\_\{\\mathrm\{complex\}\}\(F\)=\\frac\{1\}\{400\}\\sum\_\{i=1\}^\{400\}\\operatorname\{StrictEM\}\(\\hat\{z\}\_\{i\},z\_\{i\}\)\.\(10\)A prediction is correct only when all required fields, values, rankings, source references, checkpoints, and checksum constraints are satisfied\. Partially correct answers remain in the raw prediction logs but receive zero strict exact\-match credit\.

Efficiency is reported with the same trace fields for all methods\. We record token usage, model calls, executed nodes, executed edges, retries, latency, and available cost statistics\. For G\-MAS\-Complex, Table[2](https://arxiv.org/html/2609.21527#S4.T2)additionally reportsStrict​EM/K​Tokens\\mathrm\{Strict\\ EM\}/\\mathrm\{K\\ Tokens\}, the strict\-match percentage per 1K tokens\.

### A\.4Robustness Intervention Description

For every intervention, the task list, backend configuration, prompt template, answer parser, metric, routing policy, and runtime budget remain fixed unless the intervention explicitly targets the corresponding organization component\. Let

𝒞=\{𝒟𝒯,Θ,M𝒯,P\}\\mathcal\{C\}=\\\{\\mathcal\{D\}\_\{\\mathcal\{T\}\},\\Theta,M\_\{\\mathcal\{T\}\},P\\\}\(11\)denote the shared controls\. For intervention typerr, we construct

𝒪r′=ℐr​\(𝒪,Δ​𝒪r\),r∈\{str,node,info,exec\},\\mathcal\{O\}^\{\\prime\}\_\{r\}=\\mathcal\{I\}\_\{r\}\(\\mathcal\{O\},\\Delta\\mathcal\{O\}\_\{r\}\),\\qquad r\\in\\\{\\mathrm\{str\},\\mathrm\{node\},\\mathrm\{info\},\\mathrm\{exec\}\\\},\(12\)and measure

Δr​\(F,𝒯\)=s⁡\(F,𝒪r′,𝒯\)−s⁡\(F,𝒪,𝒯\)\.\\Delta\_\{r\}\(F,\\mathcal\{T\}\)=s\(F,\\mathcal\{O\}^\{\\prime\}\_\{r\},\\mathcal\{T\}\)\-s\(F,\\mathcal\{O\},\\mathcal\{T\}\)\.\(13\)A negative value indicates lower performance after modification under the shared controls\.

Table 6:Controls preserved by diagnostic interventions\.Yes means the factor is kept fixed relative to the reference run\.InterventionSamplesBackendPromptsParserBudgetRolesEdgesDegree seq\.Structure rewiringYesYesYesYesYesYesChangedYesNode removalYesYesYesYesYesChangedChangedNoInformation injectionYesYesYesYesYesYesYesYesExecution failureYesYesYesYesYesYesYesYesStructure intervention\.We evaluate the Original graph and degree\-preserving rewiring ratios of 10%, 30%, 50%, and 70%\. Directed edge pairsu→au\\\!\\to\\\!aandv→bv\\\!\\to\\\!bare proposed for the swapu→bu\\\!\\to\\\!bandv→av\\\!\\to\\\!a\. Proposals that create self\-loops, duplicate edges, or invalid role connections are rejected\. The actual ratio and random seed are recorded\.

Node intervention\.The No critic and No specialist variants remove the corresponding role nodes and all incident edges\. The Expert only variant retains expert nodes and their induced communication edges\. This intervention changes collaboration\-unit composition while keeping the task, model, prompt, parser, routing, and budget controls fixed\.

Information intervention\.For concentrationc∈\{0,\.2,\.4,\.6,\.8\}c\\in\\\{0,\.2,\.4,\.6,\.8\\\}, a fractionccof eligible upstream records or candidate items is replaced with a format\-valid but incorrect candidate and forwarded through the unchanged downstream graph\. We report task accuracy and whether final answers follow the injected candidate\. The trace further records whether the injected candidate reaches downstream units and whether it is adopted, corrected, or ignored before the final answer\.

Execution intervention\.For each failure countk∈\{0,1,2,3,4\}k\\in\\\{0,1,2,3,4\\\},kkworkers are marked unavailable at selected execution points\. The last valid shared state is recorded and the method resumes through its configured retry, fallback, or rerouting mechanism\.k=0k=0denotes no failure\.

### A\.5Evaluation Strategies

The benchmark follows four evaluation perspectives, matching the main questions in Section[4](https://arxiv.org/html/2609.21527#S4)\. Necessity compares single\-agent, ordinary MAS, and graph\-enhanced MAS under identical task and metric settings\. Effectiveness compares method behavior across the six broad\-suite domains and the four G\-MAS\-Complex tiers\. Robustness applies the four controlled interventions above to isolate communication topology, role composition, information quality, and execution recovery\. Efficiency reports quality together with token usage, model calls, latency, executed nodes and edges, and retries\.

Table 7:Evaluation strategy summary\.PerspectiveMain comparisonReported resultsNecessitySingle\-agent vs\. ordinary MAS vs\. graph\-enhanced MASMain 13\-dataset comparison and full 29\-dataset tableEffectivenessMethod behavior across domains and complex\-task tiersDomain averages, G\-MAS\-Complex tiers, and complete method rankingsRobustnessStructure, node, information, and execution interventionsPerformance changes under controlled organizational perturbationsEfficiencyTask quality relative to execution costStrict EM, tokens, latency, executed nodes, executed edges, and Strict EM/K tokens
### A\.6Experiment Environment

The execution record associates each run with its task, organization, backend configuration, and resource measurements\. All methods are executed through the same runner interface\. Each run records the dataset identifier, sample identifier, backend configuration, prompt\-template hash, organization hash, final prediction, parsed prediction, metric result, token usage, model calls, executed nodes and edges, retries, latency, and termination status\. Intervention runs additionally record the reference organization hash, intervention type, intervention parameters, changed nodes or edges when applicable, injected\-record identifiers, failed\-worker identifiers, and selected recovery action\.

Table 8:Recorded environment and run metadata\.The record links task, organization, backend, prediction, cost, and intervention fields\.CategoryRecorded fieldsTask stateDataset id, sample id, split or frozen\-list identifier, gold answer, parser versionOrganization stateMethod id, organization hash, node list, edge list, role configuration, routing policyBackend stateBackend configuration id, prompt\-template hash, runtime limits, random seed when usedPrediction stateRaw response, parsed prediction, invalid\-parse flag, metric result, final answerCost stateToken usage, model calls, executed nodes, executed edges, retries, latency, termination reasonIntervention stateReference organization hash, intervention type, changed nodes/edges, injected records, failed workers, recovery actionBefore aggregation, the audit script checks that reference and intervention runs share the same task identifiers, parser version, backend configuration, prompt template, and stopping budget\. Runs that fail these consistency checks are excluded from aggregate tables until regenerated\.

Validation protocol\.Each run produces a validation record before it enters an aggregate table\. The record includes the dataset and sample identifiers, method configuration, parser version, raw answer, parsed answer, metric value, and trace hash\. For G\-MAS\-Complex, the validator additionally distinguishes output\-format failures from content failures\. Format failures cannot be mapped toΩi\\Omega\_\{i\}, while content failures satisfy the schema but miss required values, rankings, references, checkpoints, or checksum constraints\. This separation keeps structured\-output reliability visible instead of folding every failure into a single exact\-match score\.

### A\.7Hyperparameter Settings

All comparisons freeze task samples, prompt templates, answer parsers, backend configuration, routing policy, and runtime limits within a comparison track\. The broad suite follows fixed dataset\-specific evaluation lists\. Datasets with repeated\-sampling analysis use three matched 500\-instance samples where applicable\. G\-MAS\-Complex is fixed\. Every method is evaluated on the same 400 frozen tasks\. Intervention sweeps use the discrete settings in Table[9](https://arxiv.org/html/2609.21527#A1.T9)\. No intervention changes the task text, gold answer, scoring parser, or input ordering\.

Table 9:Benchmark\-level hyperparameter and sweep settings\.ComponentSettingBroad\-suite samplingFull evaluation split where used, otherwise fixed ordered subset, with three matched 500\-instance samples for repeated\-sampling datasetsG\-MAS\-Complex sampling400 frozen tasks, 100 tasks each for Medium, Hard, Ultra, and ExtremeStructure rewiringOriginal, 10%, 30%, 50%, and 70% degree\-preserving rewiringNode interventionBase, No critic, No specialist, Expert onlyInformation interventionIncorrect\-information concentrationc∈\{0,\.2,\.4,\.6,\.8\}c\\in\\\{0,\.2,\.4,\.6,\.8\\\}Execution interventionFailed workersk∈\{0,1,2,3,4\}k\\in\\\{0,1,2,3,4\\\}Uncertainty reportingThree\-sample standard deviation where available, otherwise run\-seed standard deviation, recorded estimate, or binomial standard\-error proxy
### A\.8Uncertainty Sources

The detailed tables distinguish five statistical sources through cell\-level codes\. Table[10](https://arxiv.org/html/2609.21527#A1.T10)records the quantity attached to each score and the available repetition information\. Standard deviations, analytic standard errors, and empirical estimates retain their distinct statistical meanings\. Ranking markers order the displayed central scores, including equal numerical values\. A paired score comparison specifies the common statistic and the matched task and seed records\.

CodeCellsRecorded quantityRepetition informations\\mathrm\{s\}221Mean and sample standard deviation across sampled task sets\. Each set contains 500 instances\.Three sample setsr\\mathrm\{r\}152Fixed reference score with an attached run\-seed standard deviation\.Run count unrecordede\\mathrm\{e\}34Reference score with a recorded empirical estimate combining dataset and method components at weights 0\.70 and 0\.30\.Component counts unrecordedb\\mathrm\{b\}100Reference score with a binomial standard\-error estimate computed from that score and the displayed evaluation countNN\.Analytic estimatem\\mathrm\{m\}3Backup run\-seed mean and standard deviation for cells with an empty reference metric\.Run count unrecordedTable 10:Statistical sources and cell counts for the 510 entries in the detailed result tables\.For the three percentage scoresy1,y2,y3y\_\{1\},y\_\{2\},y\_\{3\}, codes\\mathrm\{s\}uses

y¯=13​∑j=13yj,s=12​∑j=13\(yj−y¯\)2\.\\bar\{y\}=\\frac\{1\}\{3\}\\sum\_\{j=1\}^\{3\}y\_\{j\},\\qquad s=\\sqrt\{\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{3\}\(y\_\{j\}\-\\bar\{y\}\)^\{2\}\}\.\(14\)For codeb\\mathrm\{b\}, letppbe the reported score divided by 100\. The recorded estimate in percentage points is100​p⁡\(1−p\)/N100\\sqrt\{p\(1\-p\)/N\}, using an independent binary\-outcome model\. Applying this model to CodeQ requires a binary task\-outcome definition in the scoring record\. The three codem\\mathrm\{m\}entries are DeepSeek, BigMAS, and MASRouter on MultiAgentBench\-Coding\. Source records preserve the central value and the statistical quantity as separate fields\.

### A\.9Supplementary Results

The complete 29\-dataset and 17\-baseline comparison is reported in Table[11](https://arxiv.org/html/2609.21527#A1.T11)\. The repeated\-sampling summary for the 13 matched\-sampling datasets is provided in Table[18](https://arxiv.org/html/2609.21527#A1.T18)\. These tables are separated from the dataset description to keep the appendix layout stable\.

Reading the detailed results\.Each entry reports a central score in percent, followed by an uncertainty term in percentage points and a source code\. Codes\\mathrm\{s\}gives the mean and sample standard deviation across three 500\-instance sample sets\. Coder\\mathrm\{r\}appends a recorded run\-seed standard deviation to the fixed reference score\. Codee\\mathrm\{e\}appends the recorded estimate combining dataset and method components with weights 0\.70 and 0\.30\. Codeb\\mathrm\{b\}appends a binomial standard\-error estimate using the displayedNN\. Codem\\mathrm\{m\}gives a backup run\-seed mean and standard deviation for the three cells whose reference metric is missing\. Bold and underlined scores mark the highest and second\-highest distinct values within each dataset, including ties\. Horizontal rules separate single\-agent, ordinary multi\-agent, and graph\-enhanced configurations\.

MethodGSM8KAcc\.N=1319N=1319SVAMPAcc\.N=1000N=1000MultiArithAcc\.N=600N=600AQuAAcc\.N=254N=254ASDiv\-AAcc\.N=500N=500Game\-of\-24Acc\.N=1362N=1362MATHAcc\.N=500N=500DeepSeek93\.71¯\\underline\{93\.71\}\(0\.33\)r\(0\.33\)^\{\\mathrm\{r\}\}94\.0094\.00\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.45\)r\(0\.45\)^\{\\mathrm\{r\}\}88\.5388\.53\(1\.10\)s\(1\.10\)^\{\\mathrm\{s\}\}58\.8858\.88\(1\.33\)b\(1\.33\)^\{\\mathrm\{b\}\}55\.7355\.73\(3\.64\)s\(3\.64\)^\{\\mathrm\{s\}\}Qwen59\.4459\.44\(0\.56\)e\(0\.56\)^\{\\mathrm\{e\}\}90\.9090\.90\(0\.91\)b\(0\.91\)^\{\\mathrm\{b\}\}97\.1797\.17\(0\.68\)b\(0\.68\)^\{\\mathrm\{b\}\}68\.9068\.90\(0\.86\)e\(0\.86\)^\{\\mathrm\{e\}\}86\.8786\.87\(0\.42\)s\(0\.42\)^\{\\mathrm\{s\}\}21\.0021\.00\(1\.10\)b\(1\.10\)^\{\\mathrm\{b\}\}45\.2045\.20\(2\.27\)s\(2\.27\)^\{\\mathrm\{s\}\}AutoGen92\.8792\.87\(0\.34\)r\(0\.34\)^\{\\mathrm\{r\}\}93\.7093\.70\(0\.77\)b\(0\.77\)^\{\\mathrm\{b\}\}98\.5098\.50\(0\.19\)r\(0\.19\)^\{\\mathrm\{r\}\}88\.9888\.98\(1\.72\)r\(1\.72\)^\{\\mathrm\{r\}\}88\.5388\.53\(0\.99\)s\(0\.99\)^\{\\mathrm\{s\}\}67\.91¯\\underline\{67\.91\}\(1\.26\)b\(1\.26\)^\{\\mathrm\{b\}\}56\.1356\.13\(2\.66\)s\(2\.66\)^\{\\mathrm\{s\}\}MADebate92\.8792\.87\(0\.24\)r\(0\.24\)^\{\\mathrm\{r\}\}94\.2094\.20\(0\.74\)b\(0\.74\)^\{\\mathrm\{b\}\}98\.83\\mathbf\{98\.83\}\(0\.00\)r\(0\.00\)^\{\\mathrm\{r\}\}89\.3789\.37\(0\.23\)r\(0\.23\)^\{\\mathrm\{r\}\}88\.5388\.53\(0\.99\)s\(0\.99\)^\{\\mathrm\{s\}\}62\.7062\.70\(1\.31\)b\(1\.31\)^\{\\mathrm\{b\}\}55\.2755\.27\(3\.42\)s\(3\.42\)^\{\\mathrm\{s\}\}DyLAN93\.3393\.33\(0\.41\)e\(0\.41\)^\{\\mathrm\{e\}\}94\.0094\.00\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}98\.67¯\\underline\{98\.67\}\(0\.29\)r\(0\.29\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.99\)r\(0\.99\)^\{\\mathrm\{r\}\}88\.80\\mathbf\{88\.80\}\(1\.40\)s\(1\.40\)^\{\\mathrm\{s\}\}62\.4162\.41\(1\.31\)b\(1\.31\)^\{\\mathrm\{b\}\}56\.0756\.07\(3\.24\)s\(3\.24\)^\{\\mathrm\{s\}\}GPTSwarm93\.5693\.56\(0\.43\)r\(0\.43\)^\{\\mathrm\{r\}\}94\.1094\.10\(0\.46\)r\(0\.46\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}91\.34\\mathbf\{91\.34\}\(0\.68\)e\(0\.68\)^\{\\mathrm\{e\}\}88\.6788\.67\(0\.81\)s\(0\.81\)^\{\\mathrm\{s\}\}65\.7165\.71\(0\.85\)r\(0\.85\)^\{\\mathrm\{r\}\}57\.0057\.00\(2\.84\)s\(2\.84\)^\{\\mathrm\{s\}\}GDesigner93\.6393\.63\(0\.27\)r\(0\.27\)^\{\\mathrm\{r\}\}94\.0094\.00\(0\.06\)r\(0\.06\)^\{\\mathrm\{r\}\}98\.83\\mathbf\{98\.83\}\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}88\.9888\.98\(0\.82\)r\(0\.82\)^\{\\mathrm\{r\}\}88\.6088\.60\(1\.31\)s\(1\.31\)^\{\\mathrm\{s\}\}67\.6267\.62\(1\.27\)b\(1\.27\)^\{\\mathrm\{b\}\}56\.7356\.73\(3\.37\)s\(3\.37\)^\{\\mathrm\{s\}\}SparseCT93\.86\\mathbf\{93\.86\}\(0\.41\)e\(0\.41\)^\{\\mathrm\{e\}\}93\.9093\.90\(0\.76\)b\(0\.76\)^\{\\mathrm\{b\}\}98\.67¯\\underline\{98\.67\}\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.68\)r\(0\.68\)^\{\\mathrm\{r\}\}88\.80\\mathbf\{88\.80\}\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}64\.5464\.54\(1\.30\)b\(1\.30\)^\{\\mathrm\{b\}\}57\.13¯\\underline\{57\.13\}\(3\.87\)s\(3\.87\)^\{\\mathrm\{s\}\}GraphSearch93\.5693\.56\(0\.13\)r\(0\.13\)^\{\\mathrm\{r\}\}94\.1094\.10\(0\.21\)r\(0\.21\)^\{\\mathrm\{r\}\}98\.67¯\\underline\{98\.67\}\(0\.00\)r\(0\.00\)^\{\\mathrm\{r\}\}89\.3789\.37\(1\.20\)r\(1\.20\)^\{\\mathrm\{r\}\}88\.6788\.67\(0\.90\)s\(0\.90\)^\{\\mathrm\{s\}\}67\.5567\.55\(1\.27\)b\(1\.27\)^\{\\mathrm\{b\}\}56\.4056\.40\(3\.46\)s\(3\.46\)^\{\\mathrm\{s\}\}GraphR192\.8792\.87\(0\.27\)r\(0\.27\)^\{\\mathrm\{r\}\}93\.6093\.60\(0\.55\)r\(0\.55\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}90\.55¯\\underline\{90\.55\}\(0\.23\)r\(0\.23\)^\{\\mathrm\{r\}\}88\.6088\.60\(1\.06\)s\(1\.06\)^\{\\mathrm\{s\}\}66\.8966\.89\(1\.28\)b\(1\.28\)^\{\\mathrm\{b\}\}56\.7356\.73\(2\.72\)s\(2\.72\)^\{\\mathrm\{s\}\}R\-GFM93\.1893\.18\(0\.20\)r\(0\.20\)^\{\\mathrm\{r\}\}93\.8093\.80\(0\.26\)r\(0\.26\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.91\)r\(0\.91\)^\{\\mathrm\{r\}\}88\.80\\mathbf\{88\.80\}\(1\.44\)s\(1\.44\)^\{\\mathrm\{s\}\}66\.0866\.08\(1\.28\)b\(1\.28\)^\{\\mathrm\{b\}\}55\.7355\.73\(3\.88\)s\(3\.88\)^\{\\mathrm\{s\}\}BigMAS93\.1893\.18\(0\.19\)r\(0\.19\)^\{\\mathrm\{r\}\}94\.30¯\\underline\{94\.30\}\(0\.44\)r\(0\.44\)^\{\\mathrm\{r\}\}98\.67¯\\underline\{98\.67\}\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}90\.55¯\\underline\{90\.55\}\(0\.45\)r\(0\.45\)^\{\\mathrm\{r\}\}88\.5388\.53\(1\.51\)s\(1\.51\)^\{\\mathrm\{s\}\}65\.9365\.93\(1\.28\)b\(1\.28\)^\{\\mathrm\{b\}\}57\.47\\mathbf\{57\.47\}\(3\.59\)s\(3\.59\)^\{\\mathrm\{s\}\}MASRouter93\.71¯\\underline\{93\.71\}\(0\.23\)r\(0\.23\)^\{\\mathrm\{r\}\}93\.8093\.80\(0\.17\)r\(0\.17\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}88\.9888\.98\(0\.39\)r\(0\.39\)^\{\\mathrm\{r\}\}88\.4088\.40\(1\.59\)s\(1\.59\)^\{\\mathrm\{s\}\}63\.1463\.14\(1\.31\)b\(1\.31\)^\{\\mathrm\{b\}\}56\.0056\.00\(3\.33\)s\(3\.33\)^\{\\mathrm\{s\}\}GoAgent92\.8792\.87\(0\.04\)r\(0\.04\)^\{\\mathrm\{r\}\}94\.30¯\\underline\{94\.30\}\(0\.31\)r\(0\.31\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.00\)r\(0\.00\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.79\)r\(0\.79\)^\{\\mathrm\{r\}\}88\.5388\.53\(0\.99\)s\(0\.99\)^\{\\mathrm\{s\}\}74\.08\\mathbf\{74\.08\}\(1\.19\)b\(1\.19\)^\{\\mathrm\{b\}\}56\.0756\.07\(2\.89\)s\(2\.89\)^\{\\mathrm\{s\}\}VeriMap93\.6393\.63\(0\.39\)r\(0\.39\)^\{\\mathrm\{r\}\}93\.6093\.60\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}98\.83\\mathbf\{98\.83\}\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.68\)r\(0\.68\)^\{\\mathrm\{r\}\}88\.73¯\\underline\{88\.73\}\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}65\.7165\.71\(0\.28\)r\(0\.28\)^\{\\mathrm\{r\}\}56\.3356\.33\(3\.87\)s\(3\.87\)^\{\\mathrm\{s\}\}ARGDes93\.6393\.63\(0\.38\)r\(0\.38\)^\{\\mathrm\{r\}\}94\.60\\mathbf\{94\.60\}\(0\.71\)b\(0\.71\)^\{\\mathrm\{b\}\}98\.67¯\\underline\{98\.67\}\(0\.17\)r\(0\.17\)^\{\\mathrm\{r\}\}90\.55¯\\underline\{90\.55\}\(0\.99\)r\(0\.99\)^\{\\mathrm\{r\}\}88\.5388\.53\(1\.17\)s\(1\.17\)^\{\\mathrm\{s\}\}67\.1167\.11\(1\.27\)b\(1\.27\)^\{\\mathrm\{b\}\}56\.5356\.53\(3\.01\)s\(3\.01\)^\{\\mathrm\{s\}\}EIB93\.5693\.56\(0\.08\)r\(0\.08\)^\{\\mathrm\{r\}\}94\.30¯\\underline\{94\.30\}\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}98\.5098\.50\(0\.10\)r\(0\.10\)^\{\\mathrm\{r\}\}89\.7689\.76\(0\.39\)r\(0\.39\)^\{\\mathrm\{r\}\}88\.80\\mathbf\{88\.80\}\(1\.40\)s\(1\.40\)^\{\\mathrm\{s\}\}67\.8467\.84\(1\.27\)b\(1\.27\)^\{\\mathrm\{b\}\}56\.6056\.60\(3\.29\)s\(3\.29\)^\{\\mathrm\{s\}\}Table 11:Mathematical reasoning results\. Each entry places the central score above its uncertainty term and source code\. Source codes are defined above\.MethodMMLUAcc\.N=500N=500MMLU\-ReduxAcc\.N=500N=500MMLU\-ProAcc\.N=500N=500StrategyQAAcc\.N=500N=500CommonsenseQAAcc\.N=500N=500ARC\-EasyAcc\.N=500N=500ARC\-ChallengeAcc\.N=1418N=1418DeepSeek85\.4085\.40\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}83\.7383\.73\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}73\.0773\.07\(0\.76\)s\(0\.76\)^\{\\mathrm\{s\}\}84\.6084\.60\(0\.80\)s\(0\.80\)^\{\\mathrm\{s\}\}85\.4085\.40\(0\.53\)s\(0\.53\)^\{\\mathrm\{s\}\}94\.27¯\\underline\{94\.27\}\(0\.61\)s\(0\.61\)^\{\\mathrm\{s\}\}96\.2696\.26\(0\.50\)b\(0\.50\)^\{\\mathrm\{b\}\}Qwen82\.3382\.33\(2\.76\)s\(2\.76\)^\{\\mathrm\{s\}\}79\.8079\.80\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}60\.9360\.93\(0\.90\)s\(0\.90\)^\{\\mathrm\{s\}\}79\.4079\.40\(1\.93\)s\(1\.93\)^\{\\mathrm\{s\}\}85\.6785\.67\(1\.33\)s\(1\.33\)^\{\\mathrm\{s\}\}94\.0794\.07\(0\.76\)s\(0\.76\)^\{\\mathrm\{s\}\}96\.0596\.05\(0\.52\)b\(0\.52\)^\{\\mathrm\{b\}\}AutoGen85\.8085\.80\(0\.35\)s\(0\.35\)^\{\\mathrm\{s\}\}84\.0084\.00\(1\.73\)s\(1\.73\)^\{\\mathrm\{s\}\}71\.8071\.80\(1\.20\)s\(1\.20\)^\{\\mathrm\{s\}\}84\.1384\.13\(0\.70\)s\(0\.70\)^\{\\mathrm\{s\}\}85\.6785\.67\(1\.30\)s\(1\.30\)^\{\\mathrm\{s\}\}94\.27¯\\underline\{94\.27\}\(0\.76\)s\(0\.76\)^\{\\mathrm\{s\}\}96\.6196\.61\(0\.15\)r\(0\.15\)^\{\\mathrm\{r\}\}MADebate85\.9385\.93\(0\.12\)s\(0\.12\)^\{\\mathrm\{s\}\}83\.6083\.60\(0\.53\)s\(0\.53\)^\{\\mathrm\{s\}\}71\.8771\.87\(1\.14\)s\(1\.14\)^\{\\mathrm\{s\}\}84\.0784\.07\(1\.96\)s\(1\.96\)^\{\\mathrm\{s\}\}85\.2085\.20\(1\.56\)s\(1\.56\)^\{\\mathrm\{s\}\}94\.0794\.07\(0\.81\)s\(0\.81\)^\{\\mathrm\{s\}\}96\.5496\.54\(0\.49\)b\(0\.49\)^\{\\mathrm\{b\}\}DyLAN86\.0086\.00\(1\.39\)s\(1\.39\)^\{\\mathrm\{s\}\}83\.9383\.93\(1\.50\)s\(1\.50\)^\{\\mathrm\{s\}\}73\.87\\mathbf\{73\.87\}\(1\.21\)s\(1\.21\)^\{\\mathrm\{s\}\}86\.67\\mathbf\{86\.67\}\(0\.70\)s\(0\.70\)^\{\\mathrm\{s\}\}85\.7385\.73\(0\.61\)s\(0\.61\)^\{\\mathrm\{s\}\}94\.0794\.07\(0\.76\)s\(0\.76\)^\{\\mathrm\{s\}\}96\.76¯\\underline\{96\.76\}\(0\.19\)r\(0\.19\)^\{\\mathrm\{r\}\}GPTSwarm85\.6085\.60\(0\.53\)s\(0\.53\)^\{\\mathrm\{s\}\}84\.40¯\\underline\{84\.40\}\(0\.35\)s\(0\.35\)^\{\\mathrm\{s\}\}70\.7370\.73\(0\.61\)s\(0\.61\)^\{\\mathrm\{s\}\}85\.6785\.67\(2\.84\)s\(2\.84\)^\{\\mathrm\{s\}\}85\.5385\.53\(0\.64\)s\(0\.64\)^\{\\mathrm\{s\}\}94\.27¯\\underline\{94\.27\}\(0\.76\)s\(0\.76\)^\{\\mathrm\{s\}\}96\.83\\mathbf\{96\.83\}\(0\.19\)r\(0\.19\)^\{\\mathrm\{r\}\}GDesigner85\.6785\.67\(1\.51\)s\(1\.51\)^\{\\mathrm\{s\}\}84\.2784\.27\(1\.17\)s\(1\.17\)^\{\\mathrm\{s\}\}71\.8771\.87\(0\.31\)s\(0\.31\)^\{\\mathrm\{s\}\}85\.0085\.00\(2\.03\)s\(2\.03\)^\{\\mathrm\{s\}\}86\.1386\.13\(1\.36\)s\(1\.36\)^\{\\mathrm\{s\}\}94\.2094\.20\(1\.06\)s\(1\.06\)^\{\\mathrm\{s\}\}96\.3396\.33\(0\.50\)b\(0\.50\)^\{\\mathrm\{b\}\}SparseCT85\.2085\.20\(0\.87\)s\(0\.87\)^\{\\mathrm\{s\}\}83\.7383\.73\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}71\.3371\.33\(1\.10\)s\(1\.10\)^\{\\mathrm\{s\}\}85\.8085\.80\(0\.80\)s\(0\.80\)^\{\\mathrm\{s\}\}86\.1386\.13\(1\.21\)s\(1\.21\)^\{\\mathrm\{s\}\}93\.9393\.93\(0\.64\)s\(0\.64\)^\{\\mathrm\{s\}\}96\.6996\.69\(0\.48\)b\(0\.48\)^\{\\mathrm\{b\}\}GraphSearch85\.8085\.80\(0\.80\)s\(0\.80\)^\{\\mathrm\{s\}\}84\.40¯\\underline\{84\.40\}\(1\.91\)s\(1\.91\)^\{\\mathrm\{s\}\}71\.6071\.60\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}85\.6085\.60\(0\.20\)s\(0\.20\)^\{\\mathrm\{s\}\}86\.0786\.07\(1\.27\)s\(1\.27\)^\{\\mathrm\{s\}\}94\.27¯\\underline\{94\.27\}\(0\.58\)s\(0\.58\)^\{\\mathrm\{s\}\}96\.4096\.40\(0\.49\)b\(0\.49\)^\{\\mathrm\{b\}\}GraphR185\.3385\.33\(1\.30\)s\(1\.30\)^\{\\mathrm\{s\}\}84\.0084\.00\(1\.06\)s\(1\.06\)^\{\\mathrm\{s\}\}70\.5370\.53\(1\.33\)s\(1\.33\)^\{\\mathrm\{s\}\}85\.6085\.60\(0\.69\)s\(0\.69\)^\{\\mathrm\{s\}\}85\.4785\.47\(1\.67\)s\(1\.67\)^\{\\mathrm\{s\}\}94\.2094\.20\(0\.72\)s\(0\.72\)^\{\\mathrm\{s\}\}96\.83\\mathbf\{96\.83\}\(0\.18\)r\(0\.18\)^\{\\mathrm\{r\}\}R\-GFM85\.4785\.47\(1\.27\)s\(1\.27\)^\{\\mathrm\{s\}\}84\.2784\.27\(1\.62\)s\(1\.62\)^\{\\mathrm\{s\}\}71\.5371\.53\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}85\.1385\.13\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}85\.9385\.93\(1\.30\)s\(1\.30\)^\{\\mathrm\{s\}\}93\.9393\.93\(0\.83\)s\(0\.83\)^\{\\mathrm\{s\}\}96\.76¯\\underline\{96\.76\}\(0\.36\)r\(0\.36\)^\{\\mathrm\{r\}\}BigMAS86\.0086\.00\(1\.11\)s\(1\.11\)^\{\\mathrm\{s\}\}84\.40¯\\underline\{84\.40\}\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}72\.2072\.20\(0\.20\)s\(0\.20\)^\{\\mathrm\{s\}\}85\.0785\.07\(1\.62\)s\(1\.62\)^\{\\mathrm\{s\}\}85\.8785\.87\(1\.03\)s\(1\.03\)^\{\\mathrm\{s\}\}93\.9393\.93\(1\.03\)s\(1\.03\)^\{\\mathrm\{s\}\}96\.83\\mathbf\{96\.83\}\(0\.16\)r\(0\.16\)^\{\\mathrm\{r\}\}MASRouter86\.53¯\\underline\{86\.53\}\(1\.03\)s\(1\.03\)^\{\\mathrm\{s\}\}84\.3384\.33\(0\.83\)s\(0\.83\)^\{\\mathrm\{s\}\}72\.5372\.53\(1\.81\)s\(1\.81\)^\{\\mathrm\{s\}\}85\.6785\.67\(0\.58\)s\(0\.58\)^\{\\mathrm\{s\}\}86\.0786\.07\(0\.70\)s\(0\.70\)^\{\\mathrm\{s\}\}94\.33\\mathbf\{94\.33\}\(0\.64\)s\(0\.64\)^\{\\mathrm\{s\}\}96\.6196\.61\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}GoAgent85\.9385\.93\(1\.14\)s\(1\.14\)^\{\\mathrm\{s\}\}83\.8783\.87\(1\.33\)s\(1\.33\)^\{\\mathrm\{s\}\}70\.3370\.33\(0\.50\)s\(0\.50\)^\{\\mathrm\{s\}\}85\.2785\.27\(2\.05\)s\(2\.05\)^\{\\mathrm\{s\}\}86\.47\\mathbf\{86\.47\}\(1\.45\)s\(1\.45\)^\{\\mathrm\{s\}\}94\.0094\.00\(1\.06\)s\(1\.06\)^\{\\mathrm\{s\}\}96\.76¯\\underline\{96\.76\}\(0\.18\)r\(0\.18\)^\{\\mathrm\{r\}\}VeriMap85\.8785\.87\(0\.46\)s\(0\.46\)^\{\\mathrm\{s\}\}84\.1384\.13\(1\.51\)s\(1\.51\)^\{\\mathrm\{s\}\}73\.53¯\\underline\{73\.53\}\(1\.70\)s\(1\.70\)^\{\\mathrm\{s\}\}83\.5383\.53\(1\.17\)s\(1\.17\)^\{\\mathrm\{s\}\}85\.4085\.40\(1\.00\)s\(1\.00\)^\{\\mathrm\{s\}\}93\.8793\.87\(0\.92\)s\(0\.92\)^\{\\mathrm\{s\}\}96\.76¯\\underline\{96\.76\}\(0\.12\)r\(0\.12\)^\{\\mathrm\{r\}\}ARGDes87\.27\\mathbf\{87\.27\}\(1\.10\)s\(1\.10\)^\{\\mathrm\{s\}\}83\.6083\.60\(1\.04\)s\(1\.04\)^\{\\mathrm\{s\}\}71\.4771\.47\(0\.58\)s\(0\.58\)^\{\\mathrm\{s\}\}86\.00¯\\underline\{86\.00\}\(1\.59\)s\(1\.59\)^\{\\mathrm\{s\}\}86\.20¯\\underline\{86\.20\}\(2\.31\)s\(2\.31\)^\{\\mathrm\{s\}\}94\.1394\.13\(0\.64\)s\(0\.64\)^\{\\mathrm\{s\}\}96\.76¯\\underline\{96\.76\}\(0\.25\)r\(0\.25\)^\{\\mathrm\{r\}\}EIB86\.1386\.13\(1\.67\)s\(1\.67\)^\{\\mathrm\{s\}\}84\.53\\mathbf\{84\.53\}\(1\.42\)s\(1\.42\)^\{\\mathrm\{s\}\}72\.4772\.47\(0\.12\)s\(0\.12\)^\{\\mathrm\{s\}\}85\.4085\.40\(1\.40\)s\(1\.40\)^\{\\mathrm\{s\}\}85\.7385\.73\(1\.17\)s\(1\.17\)^\{\\mathrm\{s\}\}93\.9393\.93\(0\.70\)s\(0\.70\)^\{\\mathrm\{s\}\}96\.6196\.61\(0\.22\)r\(0\.22\)^\{\\mathrm\{r\}\}Table 12:Knowledge and commonsense results\. Each entry places the central score above its uncertainty term and source code\. Source codes are defined above\.MethodHumanEvalAcc\.N=164N=164HumanEval\+\+Acc\.N=164N=164LiveCodeBenchv6Pass@11N=175N=175MultiAgentBenchCodingCodeQN=102N=102DeepSeek84\.76​\(0\.00\)r84\.76\\,\(0\.00\)^\{\\mathrm\{r\}\}80\.49​\(2\.14\)r\\mathbf\{80\.49\}\\,\(2\.14\)^\{\\mathrm\{r\}\}20\.57​\(2\.01\)r20\.57\\,\(2\.01\)^\{\\mathrm\{r\}\}41\.55​\(1\.06\)m41\.55\\,\(1\.06\)^\{\\mathrm\{m\}\}Qwen90\.24​\(1\.35\)e\\mathbf\{90\.24\}\\,\(1\.35\)^\{\\mathrm\{e\}\}76\.83​\(0\.70\)r76\.83\\,\(0\.70\)^\{\\mathrm\{r\}\}26\.86​\(1\.39\)e\\mathbf\{26\.86\}\\,\(1\.39\)^\{\\mathrm\{e\}\}60\.54​\(4\.84\)b\\mathbf\{60\.54\}\\,\(4\.84\)^\{\\mathrm\{b\}\}AutoGen81\.71​\(1\.21\)e81\.71\\,\(1\.21\)^\{\\mathrm\{e\}\}73\.78​\(3\.43\)b73\.78\\,\(3\.43\)^\{\\mathrm\{b\}\}18\.86​\(1\.25\)e18\.86\\,\(1\.25\)^\{\\mathrm\{e\}\}42\.01​\(4\.89\)b42\.01\\,\(4\.89\)^\{\\mathrm\{b\}\}MADebate83\.54​\(2\.11\)r83\.54\\,\(2\.11\)^\{\\mathrm\{r\}\}77\.44​\(2\.46\)r77\.44\\,\(2\.46\)^\{\\mathrm\{r\}\}21\.71​\(2\.88\)r21\.71\\,\(2\.88\)^\{\\mathrm\{r\}\}40\.69​\(4\.86\)b40\.69\\,\(4\.86\)^\{\\mathrm\{b\}\}DyLAN85\.98​\(1\.19\)e85\.98\\,\(1\.19\)^\{\\mathrm\{e\}\}78\.66​\(3\.20\)b78\.66\\,\(3\.20\)^\{\\mathrm\{b\}\}22\.86​\(1\.23\)e22\.86\\,\(1\.23\)^\{\\mathrm\{e\}\}37\.06​\(4\.78\)b37\.06\\,\(4\.78\)^\{\\mathrm\{b\}\}GPTSwarm84\.76​\(0\.93\)r84\.76\\,\(0\.93\)^\{\\mathrm\{r\}\}80\.49​\(5\.21\)r\\mathbf\{80\.49\}\\,\(5\.21\)^\{\\mathrm\{r\}\}19\.43​\(0\.00\)r19\.43\\,\(0\.00\)^\{\\mathrm\{r\}\}41\.91​\(4\.89\)b41\.91\\,\(4\.89\)^\{\\mathrm\{b\}\}GDesigner85\.37​\(1\.21\)e85\.37\\,\(1\.21\)^\{\\mathrm\{e\}\}76\.83​\(1\.76\)r76\.83\\,\(1\.76\)^\{\\mathrm\{r\}\}23\.43¯​\(1\.25\)e\\underline\{23\.43\}\\,\(1\.25\)^\{\\mathrm\{e\}\}39\.46​\(4\.84\)b39\.46\\,\(4\.84\)^\{\\mathrm\{b\}\}SparseCT84\.76​\(1\.83\)r84\.76\\,\(1\.83\)^\{\\mathrm\{r\}\}76\.83​\(1\.22\)r76\.83\\,\(1\.22\)^\{\\mathrm\{r\}\}21\.14​\(1\.44\)r21\.14\\,\(1\.44\)^\{\\mathrm\{r\}\}43\.82¯​\(4\.91\)b\\underline\{43\.82\}\\,\(4\.91\)^\{\\mathrm\{b\}\}GraphSearch81\.10​\(3\.06\)b81\.10\\,\(3\.06\)^\{\\mathrm\{b\}\}74\.39​\(1\.06\)r74\.39\\,\(1\.06\)^\{\\mathrm\{r\}\}22\.29​\(3\.15\)b22\.29\\,\(3\.15\)^\{\\mathrm\{b\}\}40\.64​\(4\.86\)b40\.64\\,\(4\.86\)^\{\\mathrm\{b\}\}GraphR186\.59​\(2\.66\)b86\.59\\,\(2\.66\)^\{\\mathrm\{b\}\}75\.61​\(1\.06\)r75\.61\\,\(1\.06\)^\{\\mathrm\{r\}\}20\.00​\(3\.02\)b20\.00\\,\(3\.02\)^\{\\mathrm\{b\}\}38\.09​\(4\.81\)b38\.09\\,\(4\.81\)^\{\\mathrm\{b\}\}R\-GFM89\.02​\(1\.16\)e89\.02\\,\(1\.16\)^\{\\mathrm\{e\}\}75\.61​\(2\.54\)r75\.61\\,\(2\.54\)^\{\\mathrm\{r\}\}19\.43​\(0\.87\)r19\.43\\,\(0\.87\)^\{\\mathrm\{r\}\}40\.74​\(4\.87\)b40\.74\\,\(4\.87\)^\{\\mathrm\{b\}\}BigMAS84\.76​\(0\.35\)r84\.76\\,\(0\.35\)^\{\\mathrm\{r\}\}76\.22​\(3\.32\)b76\.22\\,\(3\.32\)^\{\\mathrm\{b\}\}21\.71​\(1\.44\)r21\.71\\,\(1\.44\)^\{\\mathrm\{r\}\}37\.79​\(1\.04\)m37\.79\\,\(1\.04\)^\{\\mathrm\{m\}\}MASRouter76\.83​\(3\.29\)b76\.83\\,\(3\.29\)^\{\\mathrm\{b\}\}71\.95​\(3\.51\)b71\.95\\,\(3\.51\)^\{\\mathrm\{b\}\}20\.57​\(0\.57\)r20\.57\\,\(0\.57\)^\{\\mathrm\{r\}\}37\.45​\(2\.48\)m37\.45\\,\(2\.48\)^\{\\mathrm\{m\}\}GoAgent79\.88​\(1\.15\)e79\.88\\,\(1\.15\)^\{\\mathrm\{e\}\}74\.39​\(2\.46\)r74\.39\\,\(2\.46\)^\{\\mathrm\{r\}\}20\.57​\(1\.51\)r20\.57\\,\(1\.51\)^\{\\mathrm\{r\}\}37\.79​\(4\.80\)b37\.79\\,\(4\.80\)^\{\\mathrm\{b\}\}VeriMap78\.66​\(3\.20\)b78\.66\\,\(3\.20\)^\{\\mathrm\{b\}\}77\.44​\(3\.66\)r77\.44\\,\(3\.66\)^\{\\mathrm\{r\}\}21\.14​\(0\.33\)r21\.14\\,\(0\.33\)^\{\\mathrm\{r\}\}40\.15​\(4\.85\)b40\.15\\,\(4\.85\)^\{\\mathrm\{b\}\}ARGDes89\.63¯​\(1\.18\)e\\underline\{89\.63\}\\,\(1\.18\)^\{\\mathrm\{e\}\}79\.27¯​\(3\.17\)b\\underline\{79\.27\}\\,\(3\.17\)^\{\\mathrm\{b\}\}19\.43​\(1\.84\)r19\.43\\,\(1\.84\)^\{\\mathrm\{r\}\}39\.90​\(4\.85\)b39\.90\\,\(4\.85\)^\{\\mathrm\{b\}\}EIB83\.54​\(2\.20\)r83\.54\\,\(2\.20\)^\{\\mathrm\{r\}\}76\.83​\(1\.27\)r76\.83\\,\(1\.27\)^\{\\mathrm\{r\}\}21\.71​\(1\.44\)r21\.71\\,\(1\.44\)^\{\\mathrm\{r\}\}42\.35​\(4\.89\)b42\.35\\,\(4\.89\)^\{\\mathrm\{b\}\}Table 13:Code generation results\. Entries give score \(uncertainty\)sourcewith the defined source codes\. The recorded CodeQ metric is retained for MultiAgentBench\-Coding\.MethodPubMedQAAcc\.N=3000N=3000MedQAAcc\.N=1273N=1273MedMCQAAcc\.N=500N=500MMLU\-MedAcc\.N=1561N=1561DeepSeek81\.33​\(0\.71\)b81\.33\\,\(0\.71\)^\{\\mathrm\{b\}\}88\.45​\(0\.68\)r88\.45\\,\(0\.68\)^\{\\mathrm\{r\}\}79\.67​\(0\.99\)s79\.67\\,\(0\.99\)^\{\\mathrm\{s\}\}89\.75​\(0\.77\)b89\.75\\,\(0\.77\)^\{\\mathrm\{b\}\}Qwen80\.70​\(1\.25\)b80\.70\\,\(1\.25\)^\{\\mathrm\{b\}\}78\.95​\(0\.72\)e78\.95\\,\(0\.72\)^\{\\mathrm\{e\}\}73\.53​\(0\.61\)s73\.53\\,\(0\.61\)^\{\\mathrm\{s\}\}87\.51​\(0\.84\)b87\.51\\,\(0\.84\)^\{\\mathrm\{b\}\}AutoGen81\.43​\(0\.71\)b81\.43\\,\(0\.71\)^\{\\mathrm\{b\}\}89\.08​\(0\.55\)r89\.08\\,\(0\.55\)^\{\\mathrm\{r\}\}79\.67​\(1\.70\)s79\.67\\,\(1\.70\)^\{\\mathrm\{s\}\}89\.94​\(0\.15\)r89\.94\\,\(0\.15\)^\{\\mathrm\{r\}\}MADebate81\.87​\(0\.70\)b81\.87\\,\(0\.70\)^\{\\mathrm\{b\}\}89\.08​\(0\.53\)r89\.08\\,\(0\.53\)^\{\\mathrm\{r\}\}78\.53​\(1\.14\)s78\.53\\,\(1\.14\)^\{\\mathrm\{s\}\}90\.26¯​\(0\.75\)b\\underline\{90\.26\}\\,\(0\.75\)^\{\\mathrm\{b\}\}DyLAN82\.43​\(0\.69\)b\\mathbf\{82\.43\}\\,\(0\.69\)^\{\\mathrm\{b\}\}88\.77​\(0\.08\)r88\.77\\,\(0\.08\)^\{\\mathrm\{r\}\}79\.93​\(1\.10\)s79\.93\\,\(1\.10\)^\{\\mathrm\{s\}\}89\.56​\(0\.38\)r89\.56\\,\(0\.38\)^\{\\mathrm\{r\}\}GPTSwarm81\.67​\(0\.71\)b81\.67\\,\(0\.71\)^\{\\mathrm\{b\}\}89\.63​\(0\.54\)e89\.63\\,\(0\.54\)^\{\\mathrm\{e\}\}79\.00​\(0\.87\)s79\.00\\,\(0\.87\)^\{\\mathrm\{s\}\}89\.94​\(0\.24\)r89\.94\\,\(0\.24\)^\{\\mathrm\{r\}\}GDesigner81\.37​\(0\.71\)b81\.37\\,\(0\.71\)^\{\\mathrm\{b\}\}88\.45​\(0\.48\)r88\.45\\,\(0\.48\)^\{\\mathrm\{r\}\}79\.93​\(0\.76\)s79\.93\\,\(0\.76\)^\{\\mathrm\{s\}\}89\.69​\(0\.77\)b89\.69\\,\(0\.77\)^\{\\mathrm\{b\}\}SparseCT81\.60​\(0\.67\)r81\.60\\,\(0\.67\)^\{\\mathrm\{r\}\}89\.24​\(0\.21\)r89\.24\\,\(0\.21\)^\{\\mathrm\{r\}\}79\.67​\(1\.67\)s79\.67\\,\(1\.67\)^\{\\mathrm\{s\}\}89\.88​\(0\.39\)r89\.88\\,\(0\.39\)^\{\\mathrm\{r\}\}GraphSearch81\.53​\(0\.44\)r81\.53\\,\(0\.44\)^\{\\mathrm\{r\}\}89\.71¯​\(0\.85\)b\\underline\{89\.71\}\\,\(0\.85\)^\{\\mathrm\{b\}\}78\.73​\(1\.01\)s78\.73\\,\(1\.01\)^\{\\mathrm\{s\}\}89\.88​\(0\.27\)r89\.88\\,\(0\.27\)^\{\\mathrm\{r\}\}GraphR181\.30​\(0\.71\)b81\.30\\,\(0\.71\)^\{\\mathrm\{b\}\}88\.92​\(0\.88\)b88\.92\\,\(0\.88\)^\{\\mathrm\{b\}\}79\.13​\(0\.83\)s79\.13\\,\(0\.83\)^\{\\mathrm\{s\}\}90\.39​\(0\.75\)b\\mathbf\{90\.39\}\\,\(0\.75\)^\{\\mathrm\{b\}\}R\-GFM81\.53​\(0\.21\)r81\.53\\,\(0\.21\)^\{\\mathrm\{r\}\}89\.16​\(0\.70\)r89\.16\\,\(0\.70\)^\{\\mathrm\{r\}\}79\.53​\(0\.95\)s79\.53\\,\(0\.95\)^\{\\mathrm\{s\}\}89\.88​\(0\.13\)r89\.88\\,\(0\.13\)^\{\\mathrm\{r\}\}BigMAS81\.20​\(0\.30\)r81\.20\\,\(0\.30\)^\{\\mathrm\{r\}\}89\.87​\(0\.53\)e\\mathbf\{89\.87\}\\,\(0\.53\)^\{\\mathrm\{e\}\}78\.73​\(1\.85\)s78\.73\\,\(1\.85\)^\{\\mathrm\{s\}\}90\.07​\(0\.19\)r90\.07\\,\(0\.19\)^\{\\mathrm\{r\}\}MASRouter81\.60​\(0\.06\)r81\.60\\,\(0\.06\)^\{\\mathrm\{r\}\}89\.24​\(0\.33\)r89\.24\\,\(0\.33\)^\{\\mathrm\{r\}\}78\.80​\(1\.91\)s78\.80\\,\(1\.91\)^\{\\mathrm\{s\}\}89\.56​\(0\.10\)r89\.56\\,\(0\.10\)^\{\\mathrm\{r\}\}GoAgent80\.53​\(0\.72\)b80\.53\\,\(0\.72\)^\{\\mathrm\{b\}\}89\.40​\(0\.51\)e89\.40\\,\(0\.51\)^\{\\mathrm\{e\}\}79\.13​\(0\.92\)s79\.13\\,\(0\.92\)^\{\\mathrm\{s\}\}89\.56​\(0\.33\)r89\.56\\,\(0\.33\)^\{\\mathrm\{r\}\}VeriMap81\.27​\(0\.71\)b81\.27\\,\(0\.71\)^\{\\mathrm\{b\}\}88\.77​\(0\.52\)r88\.77\\,\(0\.52\)^\{\\mathrm\{r\}\}80\.07¯​\(1\.51\)s\\underline\{80\.07\}\\,\(1\.51\)^\{\\mathrm\{s\}\}90\.07​\(0\.11\)r90\.07\\,\(0\.11\)^\{\\mathrm\{r\}\}ARGDes82\.03¯​\(0\.70\)b\\underline\{82\.03\}\\,\(0\.70\)^\{\\mathrm\{b\}\}88\.77​\(0\.42\)r88\.77\\,\(0\.42\)^\{\\mathrm\{r\}\}80\.53​\(0\.70\)s\\mathbf\{80\.53\}\\,\(0\.70\)^\{\\mathrm\{s\}\}89\.62​\(0\.77\)b89\.62\\,\(0\.77\)^\{\\mathrm\{b\}\}EIB81\.20​\(0\.46\)r81\.20\\,\(0\.46\)^\{\\mathrm\{r\}\}89\.16​\(0\.48\)r89\.16\\,\(0\.48\)^\{\\mathrm\{r\}\}78\.87​\(1\.33\)s78\.87\\,\(1\.33\)^\{\\mathrm\{s\}\}90\.20​\(0\.75\)b90\.20\\,\(0\.75\)^\{\\mathrm\{b\}\}Table 14:Biomedical reasoning results\. Entries give score \(uncertainty\)sourcewith the defined source codes\. PubMedQA retains the recorded source sample count\.MethodTAT\-QAAcc\.N=1663N=1663ConvFinQAAcc\.N=1490N=1490FinQAAcc\.N=1147N=1147DeepSeek70\.48​\(0\.63\)r70\.48\\,\(0\.63\)^\{\\mathrm\{r\}\}60\.13​\(0\.43\)e60\.13\\,\(0\.43\)^\{\\mathrm\{e\}\}38\.10​\(0\.51\)e38\.10\\,\(0\.51\)^\{\\mathrm\{e\}\}Qwen69\.51​\(1\.13\)b69\.51\\,\(1\.13\)^\{\\mathrm\{b\}\}61\.81​\(0\.62\)e\\mathbf\{61\.81\}\\,\(0\.62\)^\{\\mathrm\{e\}\}28\.51​\(0\.70\)e28\.51\\,\(0\.70\)^\{\\mathrm\{e\}\}AutoGen70\.29​\(1\.12\)b70\.29\\,\(1\.12\)^\{\\mathrm\{b\}\}60\.74​\(0\.66\)r60\.74\\,\(0\.66\)^\{\\mathrm\{r\}\}36\.70​\(0\.57\)e36\.70\\,\(0\.57\)^\{\\mathrm\{e\}\}MADebate70\.66​\(1\.12\)b70\.66\\,\(1\.12\)^\{\\mathrm\{b\}\}61\.01​\(0\.52\)e61\.01\\,\(0\.52\)^\{\\mathrm\{e\}\}36\.97​\(0\.61\)e36\.97\\,\(0\.61\)^\{\\mathrm\{e\}\}DyLAN70\.48​\(0\.23\)r70\.48\\,\(0\.23\)^\{\\mathrm\{r\}\}60\.60​\(0\.55\)r60\.60\\,\(0\.55\)^\{\\mathrm\{r\}\}38\.88¯​\(0\.55\)e\\underline\{38\.88\}\\,\(0\.55\)^\{\\mathrm\{e\}\}GPTSwarm70\.41​\(0\.33\)r70\.41\\,\(0\.33\)^\{\\mathrm\{r\}\}60\.67​\(0\.24\)r60\.67\\,\(0\.24\)^\{\\mathrm\{r\}\}37\.66​\(1\.13\)r37\.66\\,\(1\.13\)^\{\\mathrm\{r\}\}GDesigner70\.23​\(0\.44\)r70\.23\\,\(0\.44\)^\{\\mathrm\{r\}\}61\.54¯​\(0\.48\)e\\underline\{61\.54\}\\,\(0\.48\)^\{\\mathrm\{e\}\}38\.45​\(1\.02\)r38\.45\\,\(1\.02\)^\{\\mathrm\{r\}\}SparseCT70\.41​\(0\.38\)r70\.41\\,\(0\.38\)^\{\\mathrm\{r\}\}60\.81​\(0\.47\)e60\.81\\,\(0\.47\)^\{\\mathrm\{e\}\}39\.23​\(0\.56\)e\\mathbf\{39\.23\}\\,\(0\.56\)^\{\\mathrm\{e\}\}GraphSearch71\.26​\(1\.11\)b\\mathbf\{71\.26\}\\,\(1\.11\)^\{\\mathrm\{b\}\}60\.20​\(0\.49\)r60\.20\\,\(0\.49\)^\{\\mathrm\{r\}\}38\.45​\(0\.18\)r38\.45\\,\(0\.18\)^\{\\mathrm\{r\}\}GraphR169\.75​\(1\.13\)b69\.75\\,\(1\.13\)^\{\\mathrm\{b\}\}60\.67​\(0\.34\)r60\.67\\,\(0\.34\)^\{\\mathrm\{r\}\}38\.01​\(1\.43\)b38\.01\\,\(1\.43\)^\{\\mathrm\{b\}\}R\-GFM71\.20¯​\(1\.11\)b\\underline\{71\.20\}\\,\(1\.11\)^\{\\mathrm\{b\}\}60\.40​\(0\.24\)r60\.40\\,\(0\.24\)^\{\\mathrm\{r\}\}38\.45​\(0\.46\)r38\.45\\,\(0\.46\)^\{\\mathrm\{r\}\}BigMAS70\.41​\(0\.68\)r70\.41\\,\(0\.68\)^\{\\mathrm\{r\}\}60\.40​\(0\.21\)r60\.40\\,\(0\.21\)^\{\\mathrm\{r\}\}38\.36​\(0\.52\)e38\.36\\,\(0\.52\)^\{\\mathrm\{e\}\}MASRouter69\.99​\(0\.24\)r69\.99\\,\(0\.24\)^\{\\mathrm\{r\}\}60\.20​\(0\.34\)r60\.20\\,\(0\.34\)^\{\\mathrm\{r\}\}37\.14​\(1\.43\)b37\.14\\,\(1\.43\)^\{\\mathrm\{b\}\}GoAgent71\.14​\(1\.11\)b71\.14\\,\(1\.11\)^\{\\mathrm\{b\}\}60\.74​\(0\.47\)r60\.74\\,\(0\.47\)^\{\\mathrm\{r\}\}38\.45​\(0\.13\)r38\.45\\,\(0\.13\)^\{\\mathrm\{r\}\}VeriMap70\.23​\(0\.33\)r70\.23\\,\(0\.33\)^\{\\mathrm\{r\}\}61\.28​\(1\.26\)b61\.28\\,\(1\.26\)^\{\\mathrm\{b\}\}37\.66​\(0\.83\)r37\.66\\,\(0\.83\)^\{\\mathrm\{r\}\}ARGDes71\.02​\(1\.11\)b71\.02\\,\(1\.11\)^\{\\mathrm\{b\}\}60\.60​\(0\.12\)r60\.60\\,\(0\.12\)^\{\\mathrm\{r\}\}38\.45​\(0\.31\)r38\.45\\,\(0\.31\)^\{\\mathrm\{r\}\}EIB69\.99​\(0\.67\)r69\.99\\,\(0\.67\)^\{\\mathrm\{r\}\}61\.34​\(0\.45\)e61\.34\\,\(0\.45\)^\{\\mathrm\{e\}\}38\.54​\(0\.54\)e38\.54\\,\(0\.54\)^\{\\mathrm\{e\}\}Table 15:Financial reasoning results with uncertainty terms and source codes\.MethodTabFactAcc\.N=500N=500WikiSQLAcc\.N=500N=500WikiTableQuestionsAcc\.N=500N=500SQAAcc\.N=500N=500DeepSeek92\.00​\(1\.11\)s92\.00\\,\(1\.11\)^\{\\mathrm\{s\}\}74\.07​\(0\.61\)s74\.07\\,\(0\.61\)^\{\\mathrm\{s\}\}76\.73​\(2\.16\)s76\.73\\,\(2\.16\)^\{\\mathrm\{s\}\}55\.20​\(1\.44\)s55\.20\\,\(1\.44\)^\{\\mathrm\{s\}\}Qwen81\.40​\(1\.56\)s81\.40\\,\(1\.56\)^\{\\mathrm\{s\}\}74\.93¯​\(1\.27\)s\\underline\{74\.93\}\\,\(1\.27\)^\{\\mathrm\{s\}\}65\.27​\(1\.50\)s65\.27\\,\(1\.50\)^\{\\mathrm\{s\}\}46\.53​\(2\.14\)s46\.53\\,\(2\.14\)^\{\\mathrm\{s\}\}AutoGen91\.80​\(1\.22\)s91\.80\\,\(1\.22\)^\{\\mathrm\{s\}\}74\.20​\(1\.78\)s74\.20\\,\(1\.78\)^\{\\mathrm\{s\}\}75\.93​\(1\.03\)s75\.93\\,\(1\.03\)^\{\\mathrm\{s\}\}54\.67​\(1\.97\)s54\.67\\,\(1\.97\)^\{\\mathrm\{s\}\}MADebate91\.47​\(0\.42\)s91\.47\\,\(0\.42\)^\{\\mathrm\{s\}\}74\.27​\(1\.10\)s74\.27\\,\(1\.10\)^\{\\mathrm\{s\}\}76\.67​\(1\.33\)s76\.67\\,\(1\.33\)^\{\\mathrm\{s\}\}54\.60​\(1\.93\)s54\.60\\,\(1\.93\)^\{\\mathrm\{s\}\}DyLAN92\.53​\(1\.53\)s\\mathbf\{92\.53\}\\,\(1\.53\)^\{\\mathrm\{s\}\}73\.87​\(2\.53\)s73\.87\\,\(2\.53\)^\{\\mathrm\{s\}\}77\.40¯​\(1\.44\)s\\underline\{77\.40\}\\,\(1\.44\)^\{\\mathrm\{s\}\}55\.00​\(1\.74\)s55\.00\\,\(1\.74\)^\{\\mathrm\{s\}\}GPTSwarm92\.20​\(1\.40\)s92\.20\\,\(1\.40\)^\{\\mathrm\{s\}\}74\.80​\(1\.97\)s74\.80\\,\(1\.97\)^\{\\mathrm\{s\}\}76\.60​\(1\.93\)s76\.60\\,\(1\.93\)^\{\\mathrm\{s\}\}56\.13​\(3\.13\)s56\.13\\,\(3\.13\)^\{\\mathrm\{s\}\}GDesigner91\.73​\(0\.61\)s91\.73\\,\(0\.61\)^\{\\mathrm\{s\}\}74\.93¯​\(0\.83\)s\\underline\{74\.93\}\\,\(0\.83\)^\{\\mathrm\{s\}\}76\.60​\(1\.06\)s76\.60\\,\(1\.06\)^\{\\mathrm\{s\}\}55\.73​\(4\.12\)s55\.73\\,\(4\.12\)^\{\\mathrm\{s\}\}SparseCT91\.93​\(1\.10\)s91\.93\\,\(1\.10\)^\{\\mathrm\{s\}\}74\.87​\(1\.22\)s74\.87\\,\(1\.22\)^\{\\mathrm\{s\}\}77\.47​\(0\.76\)s\\mathbf\{77\.47\}\\,\(0\.76\)^\{\\mathrm\{s\}\}56\.53​\(1\.86\)s56\.53\\,\(1\.86\)^\{\\mathrm\{s\}\}GraphSearch92\.00​\(1\.56\)s92\.00\\,\(1\.56\)^\{\\mathrm\{s\}\}74\.67​\(0\.81\)s74\.67\\,\(0\.81\)^\{\\mathrm\{s\}\}77\.13​\(1\.62\)s77\.13\\,\(1\.62\)^\{\\mathrm\{s\}\}55\.60​\(1\.44\)s55\.60\\,\(1\.44\)^\{\\mathrm\{s\}\}GraphR191\.40​\(1\.71\)s91\.40\\,\(1\.71\)^\{\\mathrm\{s\}\}74\.40​\(0\.69\)s74\.40\\,\(0\.69\)^\{\\mathrm\{s\}\}76\.80​\(0\.20\)s76\.80\\,\(0\.20\)^\{\\mathrm\{s\}\}56\.13​\(1\.81\)s56\.13\\,\(1\.81\)^\{\\mathrm\{s\}\}R\-GFM92\.13​\(2\.30\)s92\.13\\,\(2\.30\)^\{\\mathrm\{s\}\}75\.60​\(1\.25\)s\\mathbf\{75\.60\}\\,\(1\.25\)^\{\\mathrm\{s\}\}77\.40¯​\(1\.56\)s\\underline\{77\.40\}\\,\(1\.56\)^\{\\mathrm\{s\}\}56\.07​\(3\.67\)s56\.07\\,\(3\.67\)^\{\\mathrm\{s\}\}BigMAS92\.33¯​\(1\.29\)s\\underline\{92\.33\}\\,\(1\.29\)^\{\\mathrm\{s\}\}74\.73​\(0\.90\)s74\.73\\,\(0\.90\)^\{\\mathrm\{s\}\}76\.80​\(1\.11\)s76\.80\\,\(1\.11\)^\{\\mathrm\{s\}\}55\.33​\(3\.03\)s55\.33\\,\(3\.03\)^\{\\mathrm\{s\}\}MASRouter91\.40​\(1\.11\)s91\.40\\,\(1\.11\)^\{\\mathrm\{s\}\}74\.00​\(2\.55\)s74\.00\\,\(2\.55\)^\{\\mathrm\{s\}\}75\.67​\(0\.81\)s75\.67\\,\(0\.81\)^\{\\mathrm\{s\}\}57\.60¯​\(1\.51\)s\\underline\{57\.60\}\\,\(1\.51\)^\{\\mathrm\{s\}\}GoAgent92\.33¯​\(0\.81\)s\\underline\{92\.33\}\\,\(0\.81\)^\{\\mathrm\{s\}\}74\.47​\(1\.79\)s74\.47\\,\(1\.79\)^\{\\mathrm\{s\}\}76\.20​\(0\.80\)s76\.20\\,\(0\.80\)^\{\\mathrm\{s\}\}57\.73​\(2\.34\)s\\mathbf\{57\.73\}\\,\(2\.34\)^\{\\mathrm\{s\}\}VeriMap92\.07​\(1\.21\)s92\.07\\,\(1\.21\)^\{\\mathrm\{s\}\}74\.33​\(1\.63\)s74\.33\\,\(1\.63\)^\{\\mathrm\{s\}\}76\.93​\(0\.90\)s76\.93\\,\(0\.90\)^\{\\mathrm\{s\}\}54\.87​\(1\.27\)s54\.87\\,\(1\.27\)^\{\\mathrm\{s\}\}ARGDes92\.07​\(0\.83\)s92\.07\\,\(0\.83\)^\{\\mathrm\{s\}\}74\.47​\(0\.81\)s74\.47\\,\(0\.81\)^\{\\mathrm\{s\}\}77\.00​\(1\.60\)s77\.00\\,\(1\.60\)^\{\\mathrm\{s\}\}55\.27​\(1\.92\)s55\.27\\,\(1\.92\)^\{\\mathrm\{s\}\}EIB91\.40​\(0\.72\)s91\.40\\,\(0\.72\)^\{\\mathrm\{s\}\}74\.60​\(1\.59\)s74\.60\\,\(1\.59\)^\{\\mathrm\{s\}\}76\.13​\(2\.19\)s76\.13\\,\(2\.19\)^\{\\mathrm\{s\}\}55\.53​\(3\.16\)s55\.53\\,\(3\.16\)^\{\\mathrm\{s\}\}Table 16:Tabular reasoning results with uncertainty terms and source codes\.MethodG\-MAS\-ComplexStrict EMN=400N=400DeepSeek26\.75​\(2\.21\)b26\.75\\,\(2\.21\)^\{\\mathrm\{b\}\}Qwen13\.25​\(1\.70\)b13\.25\\,\(1\.70\)^\{\\mathrm\{b\}\}AutoGen23\.50​\(2\.12\)b23\.50\\,\(2\.12\)^\{\\mathrm\{b\}\}MADebate14\.00​\(1\.73\)b14\.00\\,\(1\.73\)^\{\\mathrm\{b\}\}DyLAN4\.50​\(1\.04\)b4\.50\\,\(1\.04\)^\{\\mathrm\{b\}\}GPTSwarm15\.50​\(1\.81\)b15\.50\\,\(1\.81\)^\{\\mathrm\{b\}\}GDesigner14\.50​\(1\.76\)b14\.50\\,\(1\.76\)^\{\\mathrm\{b\}\}SparseCT15\.25​\(1\.80\)b15\.25\\,\(1\.80\)^\{\\mathrm\{b\}\}GraphSearch14\.00​\(1\.73\)b14\.00\\,\(1\.73\)^\{\\mathrm\{b\}\}GraphR110\.75​\(1\.55\)b10\.75\\,\(1\.55\)^\{\\mathrm\{b\}\}R\-GFM2\.25​\(0\.74\)b2\.25\\,\(0\.74\)^\{\\mathrm\{b\}\}BigMAS19\.25​\(1\.97\)b19\.25\\,\(1\.97\)^\{\\mathrm\{b\}\}MASRouter10\.75​\(1\.55\)b10\.75\\,\(1\.55\)^\{\\mathrm\{b\}\}GoAgent50\.25​\(2\.50\)b\\mathbf\{50\.25\}\\,\(2\.50\)^\{\\mathrm\{b\}\}VeriMap29\.50¯​\(2\.28\)b\\underline\{29\.50\}\\,\(2\.28\)^\{\\mathrm\{b\}\}ARGDes13\.25​\(1\.70\)b13\.25\\,\(1\.70\)^\{\\mathrm\{b\}\}EIB14\.00​\(1\.73\)b14\.00\\,\(1\.73\)^\{\\mathrm\{b\}\}Table 17:G\-MAS\-Complex results\. Entries give strict exact match in percent with a binomial standard\-error estimate in parentheses over 400 tasks\.Sampling variability\.The following tables report means and sample standard deviations across three 500\-instance sample sets\. Their variation is across sampled task sets\. These entries reproduce the cells markeds\\mathrm\{s\}in the detailed results\. Bold and underlined means identify the highest and second\-highest distinct means in each dataset, including ties\.

MethodASDiv\-AMATHDeepSeek±1\.1088\.53\\\!\\pm\\\!1\.10±3\.6455\.73\\\!\\pm\\\!3\.64Qwen±0\.4286\.87\\\!\\pm\\\!0\.42±2\.2745\.20\\\!\\pm\\\!2\.27AutoGen±0\.9988\.53\\\!\\pm\\\!0\.99±2\.6656\.13\\\!\\pm\\\!2\.66MADebate±0\.9988\.53\\\!\\pm\\\!0\.99±3\.4255\.27\\\!\\pm\\\!3\.42DyLAN±1\.40\\mathbf\{88\.80\}\\\!\\pm\\\!1\.40±3\.2456\.07\\\!\\pm\\\!3\.24GPTSwarm±0\.8188\.67\\\!\\pm\\\!0\.81±2\.8457\.00\\\!\\pm\\\!2\.84GDesigner±1\.3188\.60\\\!\\pm\\\!1\.31±3\.3756\.73\\\!\\pm\\\!3\.37SparseCT±0\.92\\mathbf\{88\.80\}\\\!\\pm\\\!0\.9257\.13¯±3\.87\\underline\{57\.13\}\\\!\\pm\\\!3\.87GraphSearch±0\.9088\.67\\\!\\pm\\\!0\.90±3\.4656\.40\\\!\\pm\\\!3\.46GraphR1±1\.0688\.60\\\!\\pm\\\!1\.06±2\.7256\.73\\\!\\pm\\\!2\.72R\-GFM±1\.44\\mathbf\{88\.80\}\\\!\\pm\\\!1\.44±3\.8855\.73\\\!\\pm\\\!3\.88BigMAS±1\.5188\.53\\\!\\pm\\\!1\.51±3\.59\\mathbf\{57\.47\}\\\!\\pm\\\!3\.59MASRouter±1\.5988\.40\\\!\\pm\\\!1\.59±3\.3356\.00\\\!\\pm\\\!3\.33GoAgent±0\.9988\.53\\\!\\pm\\\!0\.99±2\.8956\.07\\\!\\pm\\\!2\.89VeriMap88\.73¯±0\.50\\underline\{88\.73\}\\\!\\pm\\\!0\.50±3\.8756\.33\\\!\\pm\\\!3\.87ARGDes±1\.1788\.53\\\!\\pm\\\!1\.17±3\.0156\.53\\\!\\pm\\\!3\.01EIB±1\.40\\mathbf\{88\.80\}\\\!\\pm\\\!1\.40±3\.2956\.60\\\!\\pm\\\!3\.29Table 18:Mathematical reasoning under sample\-set variation\. Entries report mean±\\pmsample standard deviation in percent across three 500\-instance sample sets\.MethodMMLUMMLU\-ReduxMMLU\-ProStrategyQACommonsenseQAARC\-EasyDeepSeek±0\.9285\.40\\\!\\pm\\\!0\.92±0\.5083\.73\\\!\\pm\\\!0\.50±0\.7673\.07\\\!\\pm\\\!0\.76±0\.8084\.60\\\!\\pm\\\!0\.80±0\.5385\.40\\\!\\pm\\\!0\.5394\.27¯±0\.61\\underline\{94\.27\}\\\!\\pm\\\!0\.61Qwen±2\.7682\.33\\\!\\pm\\\!2\.76±0\.9279\.80\\\!\\pm\\\!0\.92±0\.9060\.93\\\!\\pm\\\!0\.90±1\.9379\.40\\\!\\pm\\\!1\.93±1\.3385\.67\\\!\\pm\\\!1\.33±0\.7694\.07\\\!\\pm\\\!0\.76AutoGen±0\.3585\.80\\\!\\pm\\\!0\.35±1\.7384\.00\\\!\\pm\\\!1\.73±1\.2071\.80\\\!\\pm\\\!1\.20±0\.7084\.13\\\!\\pm\\\!0\.70±1\.3085\.67\\\!\\pm\\\!1\.3094\.27¯±0\.76\\underline\{94\.27\}\\\!\\pm\\\!0\.76MADebate±0\.1285\.93\\\!\\pm\\\!0\.12±0\.5383\.60\\\!\\pm\\\!0\.53±1\.1471\.87\\\!\\pm\\\!1\.14±1\.9684\.07\\\!\\pm\\\!1\.96±1\.5685\.20\\\!\\pm\\\!1\.56±0\.8194\.07\\\!\\pm\\\!0\.81DyLAN±1\.3986\.00\\\!\\pm\\\!1\.39±1\.5083\.93\\\!\\pm\\\!1\.50±1\.21\\mathbf\{73\.87\}\\\!\\pm\\\!1\.21±0\.70\\mathbf\{86\.67\}\\\!\\pm\\\!0\.70±0\.6185\.73\\\!\\pm\\\!0\.61±0\.7694\.07\\\!\\pm\\\!0\.76GPTSwarm±0\.5385\.60\\\!\\pm\\\!0\.5384\.40¯±0\.35\\underline\{84\.40\}\\\!\\pm\\\!0\.35±0\.6170\.73\\\!\\pm\\\!0\.61±2\.8485\.67\\\!\\pm\\\!2\.84±0\.6485\.53\\\!\\pm\\\!0\.6494\.27¯±0\.76\\underline\{94\.27\}\\\!\\pm\\\!0\.76GDesigner±1\.5185\.67\\\!\\pm\\\!1\.51±1\.1784\.27\\\!\\pm\\\!1\.17±0\.3171\.87\\\!\\pm\\\!0\.31±2\.0385\.00\\\!\\pm\\\!2\.03±1\.3686\.13\\\!\\pm\\\!1\.36±1\.0694\.20\\\!\\pm\\\!1\.06SparseCT±0\.8785\.20\\\!\\pm\\\!0\.87±0\.5083\.73\\\!\\pm\\\!0\.50±1\.1071\.33\\\!\\pm\\\!1\.10±0\.8085\.80\\\!\\pm\\\!0\.80±1\.2186\.13\\\!\\pm\\\!1\.21±0\.6493\.93\\\!\\pm\\\!0\.64GraphSearch±0\.8085\.80\\\!\\pm\\\!0\.8084\.40¯±1\.91\\underline\{84\.40\}\\\!\\pm\\\!1\.91±0\.9271\.60\\\!\\pm\\\!0\.92±0\.2085\.60\\\!\\pm\\\!0\.20±1\.2786\.07\\\!\\pm\\\!1\.2794\.27¯±0\.58\\underline\{94\.27\}\\\!\\pm\\\!0\.58GraphR1±1\.3085\.33\\\!\\pm\\\!1\.30±1\.0684\.00\\\!\\pm\\\!1\.06±1\.3370\.53\\\!\\pm\\\!1\.33±0\.6985\.60\\\!\\pm\\\!0\.69±1\.6785\.47\\\!\\pm\\\!1\.67±0\.7294\.20\\\!\\pm\\\!0\.72R\-GFM±1\.2785\.47\\\!\\pm\\\!1\.27±1\.6284\.27\\\!\\pm\\\!1\.62±0\.5071\.53\\\!\\pm\\\!0\.50±0\.5085\.13\\\!\\pm\\\!0\.50±1\.3085\.93\\\!\\pm\\\!1\.30±0\.8393\.93\\\!\\pm\\\!0\.83BigMAS±1\.1186\.00\\\!\\pm\\\!1\.1184\.40¯±0\.92\\underline\{84\.40\}\\\!\\pm\\\!0\.92±0\.2072\.20\\\!\\pm\\\!0\.20±1\.6285\.07\\\!\\pm\\\!1\.62±1\.0385\.87\\\!\\pm\\\!1\.03±1\.0393\.93\\\!\\pm\\\!1\.03MASRouter86\.53¯±1\.03\\underline\{86\.53\}\\\!\\pm\\\!1\.03±0\.8384\.33\\\!\\pm\\\!0\.83±1\.8172\.53\\\!\\pm\\\!1\.81±0\.5885\.67\\\!\\pm\\\!0\.58±0\.7086\.07\\\!\\pm\\\!0\.70±0\.64\\mathbf\{94\.33\}\\\!\\pm\\\!0\.64GoAgent±1\.1485\.93\\\!\\pm\\\!1\.14±1\.3383\.87\\\!\\pm\\\!1\.33±0\.5070\.33\\\!\\pm\\\!0\.50±2\.0585\.27\\\!\\pm\\\!2\.05±1\.45\\mathbf\{86\.47\}\\\!\\pm\\\!1\.45±1\.0694\.00\\\!\\pm\\\!1\.06VeriMap±0\.4685\.87\\\!\\pm\\\!0\.46±1\.5184\.13\\\!\\pm\\\!1\.5173\.53¯±1\.70\\underline\{73\.53\}\\\!\\pm\\\!1\.70±1\.1783\.53\\\!\\pm\\\!1\.17±1\.0085\.40\\\!\\pm\\\!1\.00±0\.9293\.87\\\!\\pm\\\!0\.92ARGDes±1\.10\\mathbf\{87\.27\}\\\!\\pm\\\!1\.10±1\.0483\.60\\\!\\pm\\\!1\.04±0\.5871\.47\\\!\\pm\\\!0\.5886\.00¯±1\.59\\underline\{86\.00\}\\\!\\pm\\\!1\.5986\.20¯±2\.31\\underline\{86\.20\}\\\!\\pm\\\!2\.31±0\.6494\.13\\\!\\pm\\\!0\.64EIB±1\.6786\.13\\\!\\pm\\\!1\.67±1\.42\\mathbf\{84\.53\}\\\!\\pm\\\!1\.42±0\.1272\.47\\\!\\pm\\\!0\.12±1\.4085\.40\\\!\\pm\\\!1\.40±1\.1785\.73\\\!\\pm\\\!1\.17±0\.7093\.93\\\!\\pm\\\!0\.70Table 19:Knowledge and commonsense under sample\-set variation\. Entries report mean±\\pmsample standard deviation in percent across three 500\-instance sample sets\.MethodMedMCQADeepSeek±0\.9979\.67\\\!\\pm\\\!0\.99Qwen±0\.6173\.53\\\!\\pm\\\!0\.61AutoGen±1\.7079\.67\\\!\\pm\\\!1\.70MADebate±1\.1478\.53\\\!\\pm\\\!1\.14DyLAN±1\.1079\.93\\\!\\pm\\\!1\.10GPTSwarm±0\.8779\.00\\\!\\pm\\\!0\.87GDesigner±0\.7679\.93\\\!\\pm\\\!0\.76SparseCT±1\.6779\.67\\\!\\pm\\\!1\.67GraphSearch±1\.0178\.73\\\!\\pm\\\!1\.01GraphR1±0\.8379\.13\\\!\\pm\\\!0\.83R\-GFM±0\.9579\.53\\\!\\pm\\\!0\.95BigMAS±1\.8578\.73\\\!\\pm\\\!1\.85MASRouter±1\.9178\.80\\\!\\pm\\\!1\.91GoAgent±0\.9279\.13\\\!\\pm\\\!0\.92VeriMap80\.07¯±1\.51\\underline\{80\.07\}\\\!\\pm\\\!1\.51ARGDes±0\.70\\mathbf\{80\.53\}\\\!\\pm\\\!0\.70EIB±1\.3378\.87\\\!\\pm\\\!1\.33Table 20:Biomedical reasoning under sample\-set variation\. Entries report mean±\\pmsample standard deviation in percent across three 500\-instance sample sets\.MethodTabFactWikiSQLWikiTableQuestionsSQADeepSeek±1\.1192\.00\\\!\\pm\\\!1\.11±0\.6174\.07\\\!\\pm\\\!0\.61±2\.1676\.73\\\!\\pm\\\!2\.16±1\.4455\.20\\\!\\pm\\\!1\.44Qwen±1\.5681\.40\\\!\\pm\\\!1\.5674\.93¯±1\.27\\underline\{74\.93\}\\\!\\pm\\\!1\.27±1\.5065\.27\\\!\\pm\\\!1\.50±2\.1446\.53\\\!\\pm\\\!2\.14AutoGen±1\.2291\.80\\\!\\pm\\\!1\.22±1\.7874\.20\\\!\\pm\\\!1\.78±1\.0375\.93\\\!\\pm\\\!1\.03±1\.9754\.67\\\!\\pm\\\!1\.97MADebate±0\.4291\.47\\\!\\pm\\\!0\.42±1\.1074\.27\\\!\\pm\\\!1\.10±1\.3376\.67\\\!\\pm\\\!1\.33±1\.9354\.60\\\!\\pm\\\!1\.93DyLAN±1\.53\\mathbf\{92\.53\}\\\!\\pm\\\!1\.53±2\.5373\.87\\\!\\pm\\\!2\.5377\.40¯±1\.44\\underline\{77\.40\}\\\!\\pm\\\!1\.44±1\.7455\.00\\\!\\pm\\\!1\.74GPTSwarm±1\.4092\.20\\\!\\pm\\\!1\.40±1\.9774\.80\\\!\\pm\\\!1\.97±1\.9376\.60\\\!\\pm\\\!1\.93±3\.1356\.13\\\!\\pm\\\!3\.13GDesigner±0\.6191\.73\\\!\\pm\\\!0\.6174\.93¯±0\.83\\underline\{74\.93\}\\\!\\pm\\\!0\.83±1\.0676\.60\\\!\\pm\\\!1\.06±4\.1255\.73\\\!\\pm\\\!4\.12SparseCT±1\.1091\.93\\\!\\pm\\\!1\.10±1\.2274\.87\\\!\\pm\\\!1\.22±0\.76\\mathbf\{77\.47\}\\\!\\pm\\\!0\.76±1\.8656\.53\\\!\\pm\\\!1\.86GraphSearch±1\.5692\.00\\\!\\pm\\\!1\.56±0\.8174\.67\\\!\\pm\\\!0\.81±1\.6277\.13\\\!\\pm\\\!1\.62±1\.4455\.60\\\!\\pm\\\!1\.44GraphR1±1\.7191\.40\\\!\\pm\\\!1\.71±0\.6974\.40\\\!\\pm\\\!0\.69±0\.2076\.80\\\!\\pm\\\!0\.20±1\.8156\.13\\\!\\pm\\\!1\.81R\-GFM±2\.3092\.13\\\!\\pm\\\!2\.30±1\.25\\mathbf\{75\.60\}\\\!\\pm\\\!1\.2577\.40¯±1\.56\\underline\{77\.40\}\\\!\\pm\\\!1\.56±3\.6756\.07\\\!\\pm\\\!3\.67BigMAS92\.33¯±1\.29\\underline\{92\.33\}\\\!\\pm\\\!1\.29±0\.9074\.73\\\!\\pm\\\!0\.90±1\.1176\.80\\\!\\pm\\\!1\.11±3\.0355\.33\\\!\\pm\\\!3\.03MASRouter±1\.1191\.40\\\!\\pm\\\!1\.11±2\.5574\.00\\\!\\pm\\\!2\.55±0\.8175\.67\\\!\\pm\\\!0\.8157\.60¯±1\.51\\underline\{57\.60\}\\\!\\pm\\\!1\.51GoAgent92\.33¯±0\.81\\underline\{92\.33\}\\\!\\pm\\\!0\.81±1\.7974\.47\\\!\\pm\\\!1\.79±0\.8076\.20\\\!\\pm\\\!0\.80±2\.34\\mathbf\{57\.73\}\\\!\\pm\\\!2\.34VeriMap±1\.2192\.07\\\!\\pm\\\!1\.21±1\.6374\.33\\\!\\pm\\\!1\.63±0\.9076\.93\\\!\\pm\\\!0\.90±1\.2754\.87\\\!\\pm\\\!1\.27ARGDes±0\.8392\.07\\\!\\pm\\\!0\.83±0\.8174\.47\\\!\\pm\\\!0\.81±1\.6077\.00\\\!\\pm\\\!1\.60±1\.9255\.27\\\!\\pm\\\!1\.92EIB±0\.7291\.40\\\!\\pm\\\!0\.72±1\.5974\.60\\\!\\pm\\\!1\.59±2\.1976\.13\\\!\\pm\\\!2\.19±3\.1655\.53\\\!\\pm\\\!3\.16Table 21:Tabular and structured reasoning under sample\-set variation\. Entries report mean±\\pmsample standard deviation in percent across three 500\-instance sample sets\.
### A\.10Artifact and Reproducibility

The benchmark artifact separates task loading, baseline construction, backend execution, answer parsing, metric calculation, cost accounting, and intervention diagnosis\. Each method is invoked through a common runner, making the raw prediction, parsed answer, metric result, trace, and cost fields comparable across single\-agent, ordinary MAS, and graph\-enhanced MAS configurations\.

Each execution trace stores a method identifier, dataset identifier, sample identifier, organization hash, prompt\-template hash, backend configuration, random seed when used, final prediction, parsed prediction, metric result, and cost fields\. For every executed unit, the trace records role name, incoming message identifiers, outgoing message identifiers, model\-call metadata, token counts, latency, retry status, and termination reason\. For intervention runs, the trace additionally stores intervention parameters and changed graph or runtime elements\.

The artifact specification includes dataset loaders, evaluation samples, baseline adapters, organization configurations, intervention operators, predictions, traces, cost logs, and aggregation scripts\.

Reproducibility records\.The artifact contains three linked record types\. A configuration record stores the dataset list, sample manifest, backend setting, prompt template, organization hash, and intervention parameters\. A scoring record stores the parser version, metric name, metric input, parsed prediction, invalid\-parse flag, and score\. An aggregation record stores the included datasets, metric weights, uncertainty source, method grouping, and the rule for selecting representative configurations\. These records make the full tables auditable and preserve dataset\-level metric identities even when results are summarized by domain or method family\.

## Appendix BSupplementary Organizational Diagnosis

### B\.1Illustrative Multi\-Document Task

This constructed example illustrates the dependency and conflict logic of G\-MAS\-Complex\. The query asks for the available inventory of product P and the identifiers of the records supporting the calculation\. The document collection contains the following records\.

RecordContentD1Initial inventory of product P is 12 units\.D2Revised inventory of product P is 9 units\. This record supersedes D1\.D3The reservation for product P is 4 units\.
The correct answer givesavailable\_unitsas 5\. It setsinventory\_sourceto D2 andreservation\_sourceto D3\. Extraction locates the records, revision checking selects D2, calculation obtains9−4=59\-4=5, and verification checks the value and source identifiers\. Two illustrative error cases distinguish these requirements\. Selecting D1 produces12−4=812\-4=8, while returning 5 without D3 omits a required source\.

A collaboration graph can assign extraction, revision checking, calculation, and integration to separate units\. The illustrative records expose the task’s dependency structure and distinguish record selection, numerical calculation, and source reporting within the answer contract\.

### B\.2Component Scoring and Input Contracts

A diagnostic extension can decompose strict exact match into the following checks\. Each component rate uses the full evaluated task list as its denominator\. The full\-task check retains all constraints of the original answer contract\.

ComponentCheckFormatThe output parses and contains the required fields and types\.Core answerRequested values and rankings agree with the reference\.SourceRequired source identifiers support the returned answer\.ConflictSelected records follow the task’s revision and authority rules\.Full taskAll required checks, checkpoints, and checksum constraints hold\.
The input contract separates the query, source documents, dependency and conflict information, and answer schema from evaluator annotations\. ForΓi\\Gamma\_\{i\}, it identifies relations stated in the input, relations encoded by document contents, and labels available to the evaluator\. Each answer field has a type, a reference value, and a validation rule\.

The construction record schema contains a task identifier, template identifier, generation procedure, and suite version\. Generation fields specify the template or model revision, prompt, sampling settings, and random seed\. Tier metadata describes document count, input length, dependency depth, conflicting records, distractors, and required answer fields\. Validation fields specify reference and checksum checks, duplicate checks within and across splits, and human review records\. Review fields identify the sampled tasks, annotators, decisions, and disagreement resolution\. A frozen manifest joins accepted task identifiers with file hashes, the scoring revision, freeze date, and license\.

### B\.3Reproducibility Record Schema

The following schema defines the information needed to reconstruct a comparison\. It specifies fields for dataset manifests, adapter configurations, and execution records\. A comparison identifier joins these sources with predictions and scores\.

Record groupRequired fieldsDatasetVersion, split, filters, sample identifiers, order, sampling seed, frozen manifest, and file hashes\.ModelExact version, request date, reasoning mode, decoding parameters, context limit, and output limit\.AdapterImplementation revision, configuration hash, retained mechanisms, changes, role prompts, and stopping rules\.LearningTraining and validation manifests, test isolation, objective, checkpoint, and graph\-generation procedure\.ToolsTool permissions, verification access, candidate selection, and format\-repair rules\.Shared stateState entries readable and writable by each unit, with message delivery and state access distinguished\.Run and scoreRun identifier, random seed, parser and metric versions, score unit, aggregation rule, and reference answers\.Cost and failureTokens, calls, retries, latency, timeouts, parsing and tool failures, termination status, concurrency, caching, and hardware\.
For candidate replacement, the record includes eligible items, selected items, replacement content, and credibility markers in both reference and modified conditions\. For worker failures, it includes the worker population, selection rule, injection times, duration, and whether the count represents simultaneous unavailability or accumulated events\. A paired record joins reference and modified executions through the same task and configuration identifiers\. It records the intervention type, selected items, random seed, schedule, and achieved modification level\. Predictions, scores, and resource records share the execution identifier\.

### B\.4Organization Analysis Protocol

Graph construction and execution describe different properties\. GPTSwarm optimizes prompts and connectivity\([Zhuge et al\., 2024](https://arxiv.org/html/2609.21527#bib.bib14)\)\. G\-Designer learns task\-dependent graph generation\([Zhang et al\., 2025](https://arxiv.org/html/2609.21527#bib.bib12)\)\. ARG\-Designer sequentially generates roles and links\([Li et al\., 2026](https://arxiv.org/html/2609.21527#bib.bib30)\)\. Configuration records can describe these properties through the following attributes\.

AttributeRecorded configurationTopologyFixed, generated for each task, or updated during execution\.LearningOptimized prompts, graph parameters, routing rules, or policies\.RolesUnit definitions and differences in prompts, tools, and models\.VerificationChecks available to each unit and their invocation rules\.RecoveryRetry, fallback, or rerouting rules and termination conditions\.
Graph records associate task identifiers with initial nodes, directed edges, updates, and executed messages\. For a simple directed graph withn≥2n\\geq 2nodes andmmedges, density ism/\[n⁡\(n−1\)\]m/\[n\(n\-1\)\]\. Letℛ\\mathcal\{R\}contain reachable ordered pairs of distinct nodes\. The reachable\-pair fraction and average reachable path length are

RG=\|ℛ\|n⁡\(n−1\),LG=1\|ℛ\|​∑\(u,v\)∈ℛdG​\(u,v\)\.R\_\{G\}=\\frac\{\|\\mathcal\{R\}\|\}\{n\(n\-1\)\},\\qquad L\_\{G\}=\\frac\{1\}\{\|\\mathcal\{R\}\|\}\\sum\_\{\(u,v\)\\in\\mathcal\{R\}\}d\_\{G\}\(u,v\)\.\(15\)HeredGd\_\{G\}is shortest directed path length, andLGL\_\{G\}requires\|ℛ\|\>0\|\\mathcal\{R\}\|\>0\. Comparing constructed edges with executed message edges connects available routes with their use\. For executions containing messages, letpvp\_\{v\}be the fraction originating from nodevv\. Normalized source entropy is

HG=−1log⁡n∑v∈Vpvlogpv,H\_\{G\}=\-\\frac\{1\}\{\\log n\}\\sum\_\{v\\in V\}p\_\{v\}\\log p\_\{v\},\(16\)with zero\-probability terms contributing zero\. Role\-level message counts and temporal records describe communication before and after an intervention\.

### B\.5Reimplementation Validation Protocol

Reimplementation validation aligns the source revision, task split, backend, prompts, graph construction, optimization, decoding, stopping rule, and metric with the original method\. Its comparison record contains the published score, reproduced score, their difference, and repetition count\. A subsequent benchmark evaluation applies the shared task presentation and scoring controls\. Method\-specific roles and coordination rules remain part of the adapter description\. This sequence connects original\-setting validation with evaluation under the benchmark configuration\.

### B\.6Follow\-up Comparisons and Statistical Records

Future budget\-matched comparisons can evaluate direct single\-agent execution, iterative planning and verification, independent candidate generation, and representative multi\-agent configurations under common model, tool, input, and resource limits\. Actual consumption accompanies task scores\. Candidate selection uses information available to the evaluated system\. Role\-replacement comparisons can preserve nodes and connections while replacing specialist prompts with general roles or exchanging role prompts\. These settings test role definitions\.

Further rewiring comparisons can check output reachability, scheduling validity, and schema compatibility for each proposed organization, alongside degree preservation\. Repeated graph samples record seeds, accepted swaps, actual replacement ratios, and execution costs\. Recovery comparisons can apply an identical failure schedule to the same configuration with recovery enabled and disabled, recording accuracy, retries, tokens, and latency\.

Paired statistical analysis resamples common task identifiers and computes score differences within each replicate\. The resampling specification groups generated tasks by template family to account for dependence among related instances\. The analysis record specifies the resampling unit, repetition count, interval method, and central estimate\. Separate run identifiers track execution randomness, while sample manifests track variation from dataset selection\.

## References

- Chenet al\.\(2026\)H\. Chen, X\. Zheng, Y\. Liu, P\. Jiao, S\. Li, H\. Liu, Z\. Zhao, Z\. Xu, I\. Khalil, and S\. PanGoAgent: group\-of\-agents communication topology generation for llm\-based multi\-agent systems\.External Links:2603\.19677,[Link](https://arxiv.org/abs/2603.19677)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.6.2.1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx3.p2.1)\.
- Chenet al\.\(2021a\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Chenet al\.\(2020\)W\. Chen, H\. Wang, J\. Chen, Y\. Zhang, H\. Wang, S\. Li, X\. Zhou, and W\. Y\. WangTabFact: a large\-scale dataset for table\-based fact verification\.External Links:1909\.02164,[Link](https://arxiv.org/abs/1909.02164)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Chenet al\.\(2021b\)Z\. Chen, W\. Chen, C\. Smiley, S\. Shah, I\. Borova, D\. Langdon, R\. Moussa, M\. Beane, T\. Huang, B\. Routledge, and W\. Y\. WangFinQA: a dataset of numerical reasoning over financial data\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 3697–3711\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.300/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.300)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Chenet al\.\(2022\)Z\. Chen, S\. Li, C\. Smiley, Z\. Ma, S\. Shah, and W\. Y\. WangConvFinQA: exploring the chain of numerical reasoning in conversational finance question answering\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 6279–6292\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.421/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.421)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.CoRRabs/2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168),2110\.14168Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.2.2.1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p2.1)\.
- Duet al\.\(2023\)Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. MordatchImproving factuality and reasoning in language models through multiagent debate\.External Links:2305\.14325,[Link](https://arxiv.org/abs/2305.14325)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.3.2.1.1),[§1](https://arxiv.org/html/2609.21527#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p2.1)\.
- Gemaet al\.\(2025\)A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani, C\. Barale, R\. McHardy, J\. Harris, J\. Kaddour, E\. van Krieken, and P\. MinerviniAre we done with mmlu?\.External Links:2406\.04127,[Link](https://arxiv.org/abs/2406.04127)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Gevaet al\.\(2021\)M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. BerantDid aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.External Links:[Link](https://aclanthology.org/2021.tacl-1.21/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00370)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Haoet al\.\(2026\)G\. Hao, Y\. Dai, X\. Qin, and S\. YuBrain\-inspired graph multi\-agent systems for llm reasoning\.External Links:2603\.15371,[Link](https://arxiv.org/abs/2603.15371)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.5.2.1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.External Links:2009\.03300,[Link](https://arxiv.org/abs/2009.03300)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.CoRRabs/2103\.03874\.External Links:[Link](https://arxiv.org/abs/2103.03874),2103\.03874Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Iyyeret al\.\(2017\)M\. Iyyer, W\. Yih, and M\. ChangSearch\-based neural structures for sequential question answering\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics,pp\. 209–220\.External Links:[Link](https://aclanthology.org/P17-1020/),[Document](https://dx.doi.org/10.18653/v1/P17-1020)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Jainet al\.\(2024\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.External Links:2403\.07974,[Link](https://arxiv.org/abs/2403.07974)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Jinet al\.\(2021\)D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. SzolovitsWhat disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.External Links:[Document](https://dx.doi.org/10.3390/app11146421),[Link](https://www.mdpi.com/2076-3417/11/14/6421)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Jinet al\.\(2019\)Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. LuPubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2567–2577\.External Links:[Link](https://aclanthology.org/D19-1259/),[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Liet al\.\(2026\)S\. Li, Y\. Liu, Q\. Wen, C\. Zhang, and S\. PanAssemble your crew: automatic multi\-agent communication topology design via autoregressive graph generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 23142–23150\.Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.5.2.1.1),[§B\.4](https://arxiv.org/html/2609.21527#A2.SS4.p1.1)\.
- Liet al\.\(2024\)Y\. Li, Y\. Du, J\. Zhang, L\. Hou, P\. Grabowski, Y\. Li, and E\. IeImproving multi\-agent debate with sparse communication topology\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7281–7294\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.427/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.427)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.5.2.1.1)\.
- Linget al\.\(2017\)W\. Ling, D\. Yogatama, C\. Dyer, and P\. BlunsomProgram induction by rationale generation: learning to solve and explain algebraic word problems\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 158–167\.External Links:[Link](https://aclanthology.org/P17-1015/),[Document](https://dx.doi.org/10.18653/v1/P17-1015)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Liuet al\.\(2026a\)H\. Liu, Z\. Ding, and X\. XieLearning graph foundation models on riemannian graph\-of\-graphs\.External Links:2605\.09993,[Link](https://arxiv.org/abs/2605.09993)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.5.2.1.1)\.
- Liuet al\.\(2026b\)J\. Liu, Y\. Sun, D\. Fan, and Q\. TanGraphSearch: agentic search\-augmented reasoning for zero\-shot graph learning\.External Links:2601\.08621,[Link](https://arxiv.org/abs/2601.08621)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.4.2.1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.External Links:2305\.01210,[Link](https://arxiv.org/abs/2305.01210)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Liuet al\.\(2024\)Z\. Liu, Y\. Zhang, P\. Li, Y\. Liu, and D\. YangA dynamic llm\-powered agent network for task\-oriented agent collaboration\.External Links:2310\.02170,[Link](https://arxiv.org/abs/2310.02170)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.3.2.1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p2.1)\.
- Luoet al\.\(2026\)H\. Luo, H\. E, G\. Chen, Q\. Lin, Y\. Guo, F\. Xu, Z\. Kuang, M\. Song, X\. Wu, Y\. Zhu, and L\. A\. TuanGraph\-r1: towards agentic graphrag framework via end\-to\-end reinforcement learning\.External Links:2507\.21892,[Link](https://arxiv.org/abs/2507.21892)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.4.2.1.1)\.
- Miaoet al\.\(2020\)S\. Miao, C\. Liang, and K\. SuA diverse corpus for evaluating and developing English math word problem solvers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 975–984\.External Links:[Link](https://aclanthology.org/2020.acl-main.92/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.92)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Palet al\.\(2022\)A\. Pal, L\. K\. Umapathi, and M\. SankarasubbuMedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,G\. Flores, G\. H\. Chen, T\. Pollard, J\. C\. Ho, and T\. Naumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.External Links:[Link](https://proceedings.mlr.press/v174/pal22a.html)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Pasupat and Liang \(2015\)P\. Pasupat and P\. LiangCompositional semantic parsing on semi\-structured tables\.InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong and M\. Strube \(Eds\.\),Beijing, China,pp\. 1470–1480\.External Links:[Link](https://aclanthology.org/P15-1142/),[Document](https://dx.doi.org/10.3115/v1/P15-1142)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Patelet al\.\(2021\)A\. Patel, S\. Bhattamishra, and N\. GoyalAre NLP models really able to solve simple math word problems?\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 2080–2094\.External Links:[Link](https://aclanthology.org/2021.naacl-main.168/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.168)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.2.2.1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p2.1)\.
- Roy and Roth \(2015\)S\. Roy and D\. RothSolving general arithmetic word problems\.InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,L\. Màrquez, C\. Callison\-Burch, and J\. Su \(Eds\.\),Lisbon, Portugal,pp\. 1743–1752\.External Links:[Link](https://aclanthology.org/D15-1202/),[Document](https://dx.doi.org/10.18653/v1/D15-1202)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Shenet al\.\(2025\)X\. Shen, Y\. Liu, Y\. Dai, Y\. Wang, R\. Miao, Y\. Tan, S\. Pan, and X\. WangUnderstanding the information propagation effects of communication topologies in llm\-based multi\-agent systems\.External Links:2505\.23352,[Link](https://arxiv.org/abs/2505.23352)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.5.2.1.1),[§1](https://arxiv.org/html/2609.21527#S1.p2.1)\.
- Talmoret al\.\(2019\)A\. Talmor, J\. Herzig, N\. Lourie, and J\. BerantCommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.External Links:2406\.01574,[Link](https://arxiv.org/abs/2406.01574)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx1.p1.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen llm applications via multi\-agent conversation\.External Links:2308\.08155,[Link](https://arxiv.org/abs/2308.08155)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.3.2.1.1),[§1](https://arxiv.org/html/2609.21527#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p2.1)\.
- Xuet al\.\(2025\)T\. Xu, D\. Zhang, K\. Mitra, and E\. HruschkaVerification\-aware planning for multi\-agent systems\.External Links:2510\.17109,[Link](https://arxiv.org/abs/2510.17109)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.6.2.1.1),[§4](https://arxiv.org/html/2609.21527#S4.SSx3.p2.1)\.
- Yaoet al\.\(2023\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.External Links:2305\.10601,[Link](https://arxiv.org/abs/2305.10601)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, G\. Zhang, B\. Liu, G\. Wan, K\. Wang, D\. Cheng, and Y\. QiMasRouter: learning to route llms for multi\-agent systems\.External Links:2502\.11133,[Link](https://arxiv.org/abs/2502.11133)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.4.2.1.1)\.
- Zhanget al\.\(2025\)G\. Zhang, Y\. Yue, X\. Sun, G\. Wan, M\. Yu, J\. Fang, K\. Wang, T\. Chen, and D\. ChengG\-designer: architecting multi\-agent communication topologies via graph neural networks\.External Links:2410\.11782,[Link](https://arxiv.org/abs/2410.11782)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.4.2.1.1),[§B\.4](https://arxiv.org/html/2609.21527#A2.SS4.p1.1),[§1](https://arxiv.org/html/2609.21527#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p4.1)\.
- Zhonget al\.\(2017\)V\. Zhong, C\. Xiong, and R\. SocherSeq2SQL: generating structured queries from natural language using reinforcement learning\.External Links:1709\.00103,[Link](https://arxiv.org/abs/1709.00103)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Zhuet al\.\(2021\)F\. Zhu, W\. Lei, Y\. Huang, C\. Wang, S\. Zhang, J\. Lv, F\. Feng, and T\. ChuaTAT\-QA: a question answering benchmark on a hybrid of tabular and textual content in finance\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3277–3287\.External Links:[Link](https://aclanthology.org/2021.acl-long.254/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.254)Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1)\.
- Zhuet al\.\(2025\)K\. Zhu, H\. Du, Z\. Hong, X\. Yang, S\. Guo, Z\. Wang, Z\. Wang, C\. Qian, X\. Tang, H\. Ji, and J\. YouMultiAgentBench : evaluating the collaboration and competition of LLM agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 8580–8622\.External Links:[Link](https://aclanthology.org/2025.acl-long.421/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.421),ISBN 979\-8\-89176\-251\-0Cited by:[§A\.1](https://arxiv.org/html/2609.21527#A1.SS1.p1.1),[§1](https://arxiv.org/html/2609.21527#S1.p2.1)\.
- Zhugeet al\.\(2024\)M\. Zhuge, W\. Wang, L\. Kirsch, F\. Faccio, D\. Khizbullin, and J\. SchmidhuberLanguage agents as optimizable graphs\.External Links:2402\.16823,[Link](https://arxiv.org/abs/2402.16823)Cited by:[Table 5](https://arxiv.org/html/2609.21527#A1.T5.4.4.2.1.1),[§B\.4](https://arxiv.org/html/2609.21527#A2.SS4.p1.1),[§1](https://arxiv.org/html/2609.21527#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.21527#S3.SS2.p4.1)\.