The Illusion of Multi-Agent Advantage

# The Illusion of Multi-Agent Advantage
Source: [https://arxiv.org/html/2606.13003](https://arxiv.org/html/2606.13003)
Prathyusha Jwalapuram1,∗Hehai Lin2,∗Chuyuan Li3Fangkai Jiao4 Sudong Wang2Yifei Ming1Zixuan Ke1Chengwei Qin2 Giuseppe Carenini3Shafiq Joty1

∗Equal contribution 1Salesforce Research2HKUST \(Guangzhou\) 3University of British Columbia4Nanyang Technological University

###### Abstract

Prevailing wisdom posits that Multi\-Agent Systems \(MAS\) are superior to Single\-Agent Systems \(SAS\), citing advantages like context protection, parallel processing and distributed decision\-making\. However, empirical support for this claim relies primarily on comparisons with SAS baselines using benchmarks that prioritize isolated reasoning tasks, which do not adequately assess these advantages\. Focusing on automatically generated MAS that are designed for enhanced generalizability over manually\-designed counterparts, we perform a rigorous, systematic evaluation against SAS, specifically Chain\-of\-Thought with Self\-Consistency \(CoT\-SC\)\. Across traditional reasoning datasets and tasks with interactive multi\-step workflows \(e\.g\.,BrowseComp\-Plus\), we demonstrate that automatic MAS consistently underperform CoT\-SC despite being up to 10x more expensive\. To isolate these failures from limitations inherent to task structure, we introduce a diagnostic synthetic dataset tailored for MAS featuring explicit task decomposition, context separation and parallelization potential\. We show that expert\-architected MAS consistently outperforms automatically generated architectures in both raw performance and cost\-efficiency on this dataset, demonstrating that existing evaluation frameworks mask critical architectural gaps and inefficiencies of complex MAS by failing to account for the marginal utility of increased computational cost\. Critically, systematic deconstruction of the generated MAS architectures reveals that current automated design paradigms produce architectural bloat that prioritizes superficial complexity which does not translate into functional utility, exposing a fundamental misalignment with multi\-agent principles\.111The dataset and code can be found at[https://multi\-agent\-eval\.github\.io/](https://multi-agent-eval.github.io/)\.

## 1Introduction

![[Uncaptioned image]](https://arxiv.org/html/2606.13003v1/img/plot_teaser_notitle.png)

Figure 1:The Illusion of Multi\-Agent Advantage\.Theory promises specialization \(left\); reality reveals redundancy and functional collapse \(right\)\. Automated frameworks often incur≈10×\\approx 10\\timesthe cost of CoT\-SC for negligible gains \(Section[4](https://arxiv.org/html/2606.13003#S4)\)\.
Although Large Language Models \(LLMs\) have evolved significantly in their capabilities, they alone as Single\-Agent Systems \(SAS\) still fall short on several complex reasoning tasks, such as BrowseComp\-Plus\(Chenet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib66)\)and Humanity’s Last Exam \(HLE\)\(Phanet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib340)\)\. Multi\-Agent Systems \(MAS\) have increasingly been introduced as a solutionKeet al\.\([2025a](https://arxiv.org/html/2606.13003#bib.bib208)\), under the assumption that multiple coordinated LLM agents would outperform SAS by enabling collective decision making through mechanisms such as task decomposition, parallel execution, context separation, role specialization, debate, reconciliation, and cross\-verification\(Foersteret al\.,[2016](https://arxiv.org/html/2606.13003#bib.bib121); Hernandez\-Lealet al\.,[2018](https://arxiv.org/html/2606.13003#bib.bib166); Wanget al\.,[2019](https://arxiv.org/html/2606.13003#bib.bib474),[2024](https://arxiv.org/html/2606.13003#bib.bib487); Zhouet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib616); Gaoet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib130)\)\.

This expectation has led to the rapid development ofautomatic MAS, characterized by an automated coordination layer that dynamically decomposes tasks, configures agent roles, routes information, and manages execution flow\(Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207); Huet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib174); Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589); Liuet al\.,[2024b](https://arxiv.org/html/2606.13003#bib.bib267); Zhanget al\.,[2025a](https://arxiv.org/html/2606.13003#bib.bib598); Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\), in contrast to manually\-designed MAS, which rely on substantial human effort and often lack generalizability to novel tasks\. Automatic MAS can also be designed as decentralized agent\-team systems, where agents communicate and act more independently\(Anthropic,[2026b](https://arxiv.org/html/2606.13003#bib.bib18); OpenClaw Agents,[2026](https://arxiv.org/html/2606.13003#bib.bib328); MiroFish,[2026](https://arxiv.org/html/2606.13003#bib.bib308)\)\. While decentralized systems offer alternative scaling properties, centralized coordination paradigms represent the current standard for high\-precision task execution and are thus the focus of our evaluation\.

Despite their popularity, the realized advantages of automatic MAS remain unclear\. Most evaluations compare MAS against SAS baselines such as Chain\-of\-Thought \(CoT\)\(Weiet al\.,[2022](https://arxiv.org/html/2606.13003#bib.bib498)\), CoT with Self\-Consistency \(CoT\-SC\)\(Wanget al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib478)\), or self\-refinement\(Madaanet al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib295)\), reporting improved accuracy in tasks such as mathematical reasoning\(Chenet al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib55); Zhouet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib616); Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207); Huet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib174); Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105); Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589),[a](https://arxiv.org/html/2606.13003#bib.bib598)\), question answering \(QA\)\(Zhouet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib616); Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207); Huet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib174); Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105); Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589)\), and coding\(Zhanget al\.,[2025d](https://arxiv.org/html/2606.13003#bib.bib601); Chenet al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib55); Zhouet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib616); Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207); Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105); Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589),[a](https://arxiv.org/html/2606.13003#bib.bib598)\)\. However, these comparisons rarely control for inference budgets such as number of LLM calls, total cost, retries, or sampled paths\. Thus, MAS may appear stronger due to increased test\-time computation rather than superior coordination\(Kapooret al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib201); Anthropic,[2025](https://arxiv.org/html/2606.13003#bib.bib16)\)\. Recent studies further question MAS robustness, showing inconsistent performance against strong SAS baselines\(Gaoet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib130)\)and instability in debate or verification mechanisms\(Wynnet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib515); Venkataramaniet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib461)\)\. While controlled analyses suggest gains depend on task topology and cost\(Kimet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib220); Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\), existing evaluations focus on outcome accuracy rather than whether motivating mechanisms \- such as task decomposition, parallelization, or context separation \- actually manifest in automated workflows\.

Moreover, AnthropicAnthropic \([2026a](https://arxiv.org/html/2606.13003#bib.bib17)\)recommends building MAS for tasks where context separation provides clear benefits like a\) protection of context, b\) parallelization and c\) specialization in terms of domain, system prompt, tool set, etc\. Although recent datasets such as MASBench\(Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\)provide a controlled framework for analyzing MAS behavior, most evaluations still rely on tasks originally designed for SAS, which do not isolate properties such as sub\-task structure, parallel execution, or role specialization\.Kimet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib220)\)categorize commonly used reasoning and QA benchmarks such as GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2606.13003#bib.bib72)\)and MMLUHendryckset al\.\([2021](https://arxiv.org/html/2606.13003#bib.bib164)\)as unsuitable for evaluating agentic capabilities, since they evaluate static reasoning, contrasting them against benchmarks such as BrowseComp\-PlusChenet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib66)\), which requires dynamic and progressive information seeking and reasoning\. To address this apparent misalignment in current MAS evaluation paradigms, our work centers on three primary investigations:

1. 1\.Comparative Performance:Do automatic MAS provide consistent and cost\-effective performance advantages over strong SAS baselines?
2. 2\.Isolating Task Suitability:When provided with explicit structural opportunities for multi\-agent execution, can automated orchestrators translate these into functional utility?
3. 3\.Architectural Alignment:Do automated systems successfully manifest core MAS principles like parallelization, specialization and context protection?

We measurecomparative performanceby conducting systematic evaluations of automatic MAS against strong SAS baselines, particularly CoT\-SC\. Our evaluation spans multiple model sizes and families, including GPT\-4o, GPT\-5, GPT\-OSS\-120B, and Gemini\-2\.5\-Pro, and covers both standard reasoning tasks and more complex agentic settings such as GPQA\-Diamond, HLE\-Maths, SWE\-Bench Lite, and BrowseComp\-Plus\. We find that automatic MAS do not consistently outperform SAS; in many settings, CoT\-SC matches or exceeds MAS performance while being more cost\-efficient\.

Toisolate task suitabilityas a contributing factor and to investigate if MAS principles emerge under favorable conditions, we introduce theSynthetic Multi\-Hop Financial Reasoning\(SMFR\) dataset\.SMFRfeatures an explicit sub\-task structure, context\-heavy inputs, and clear opportunities for parallelization and specialization\. We find once again that CoT\-SC reliably outperforms automatic MAS on this task, demonstrating that task suitability is not a factor in their poor performance\. We also construct an expert\-designed MAS baseline with explicit decomposition, role specialization, and deterministic orchestration\. This baseline performs strongly, demonstrating that tasks can benefit from MAS when the system is properly structured\.

We further analyze thearchitectural alignmentof the generated MAS with core MAS principles\. Our deconstruction of these workflows showsarchitectural bloat and systematic failure in core agentic functions\. Specifically: \(i\) assigned agent roles are often functionally redundant; \(ii\) many automated MAS effectively collapse into basic CoT\-SC execution; and crucially \(iii\) this lack of specialization is consistent across disparate tasks, exposing a fundamental deficit in adaptive task decomposition\. Together, our findings suggest that the perceived advantage of automated MAS is often a byproduct of superficial complexity rather than structural synergy\. Our contributions include:

1. 1\.A Critical Re\-evaluation of the MAS Advantage:We demonstrate through systematic benchmarking that automated MAS rarely outperform SAS baselines when accounting for cost\-efficiency and baseline strength\.
2. 2\.TheSMFRDiagnostic Benchmark: We introduceSynthetic Multi\-Hop Financial Reasoning, a diagnostic task featuring explicit sub\-structures and a gold\-standardExpert\-MASto establish an empirical performance upper bound for MAS\.
3. 3\.Architectural Deconstruction:We provide a rigorous analysis of synthesized MAS workflows, exposing functional collapse where complex automated designs revert to basic single\-agent execution in practice\.

## 2Related Work

While “agenticness” exists on a continuum\(Kapooret al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib201)\), we distinguish Single\-Agent Systems \(SAS\) from Multi\-Agent Systems \(MAS\) based on the locus of reasoning\. We define SAS as a single sequential control loop governed by one LLM instance, encompassing tool use\(Yaoet al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib547)\), self\-reflection\(Madaanet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib296)\), and CoT reasoning\. In contrast, MAS features multiple LLM\-backed agents interacting through structured protocols\(Xiet al\.,[2023](https://arxiv.org/html/2606.13003#bib.bib516)\), where behavior emerges from collective reasoning\. Our work specifically evaluates centralized, automated MAS, where an orchestrator dynamically manages roles and information flow, as these frameworks represent the current frontier of agentic scaling\.

Inference\-time Automatic MAS\.These MAS adapt the agent configuration dynamically for each query\. DyLAN\(Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105)\)utilizes importance scoring to select sub\-agents on the fly, while MAS\-Zero\(Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207)\)attempts zero\-shot coordination without external validation\.

Optimized Automatic MAS\.To minimize test\-time overhead, these frameworks discover or train optimal architectures prior to deployment\. ADAS\(Huet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib177)\)and AFlow\(Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589)\)treat MAS design as a code\-generation task, utilizing Monte Carlo Tree Search \(MCTS\) to find workflows that perform well on a validation set\. Others, such as ToolOrchestra\(Suet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib425)\)and MAS\-Orchestra\(Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\), use Reinforcement Learning \(RL\) to train a centralized orchestrator\. Frameworks like MaAS\(Zhanget al\.,[2025a](https://arxiv.org/html/2606.13003#bib.bib598)\)occupy a middle ground; while the underlying operator distributions are pre\-optimized, the system performs inference\-time routing by sampling query\-dependent architectures on the fly\.

We evaluate both kinds of systems to determine if dynamic flexibility or pre\-optimized workflows justify the significant per\-query compute overhead without architectural bloat and functional collapse\.

Diagnostics of Multi\-Agent Failure\.Cemriet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib45)\)categorize execution\-level failures \(e\.g\.,communication lapses\), whereas we diagnose structural inefficiencies \(e\.g\.,role redundancy, functional collapse\), inherent to automated MAS search\. WhileKapooret al\.\([2024](https://arxiv.org/html/2606.13003#bib.bib201)\)andKimet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib220)\)question benchmark suitability for MAS, we introduceSMFRas a diagnostic tool to isolate task suitability\. Finally, addressingTran and Kiela \([2026](https://arxiv.org/html/2606.13003#bib.bib452)\)’s critique regarding compute\-confounded gains, we show that CoT\-SC consistently outperforms MAS despite a significantly lower token budget\. This indicates that current automated designs suffer from architectural bloat, failing to translate high expenditure into reasoning gains\.

## 3Critical Re\-Evaluation of the MAS Advantage

To investigate whether automatic MAS show consistent and cost\-effective performance advantages over strong SAS baselines, we conduct a large\-scale audit comparing SAS and MAS performance and cost across multiple LLM model sizes and families, covering standard reasoning tasks and complex agentic settings\. We specifically test the hypothesis that MAS provide a superior scaling path compared to simple, budget\-matched ensembling\.

### 3\.1Experimental Setup

Benchmark Datasets\.As detailed in Section[1](https://arxiv.org/html/2606.13003#S1), mathematical reasoning, QA, and coding are the primary domains for evaluating MAS\. Following standard practice, we select the most up\-to\-date and challenging variants of these tasks to attempt a systematic reproduction of commonly reported MAS improvements, ensuring our evaluation reflects the current performance ceiling of the field\. Specifically, we target: \(i\) mathematical reasoning throughHLE\-MathsPhanet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib340)\); \(ii\) QA throughGPQA\-DiamondReinet al\.\([2023](https://arxiv.org/html/2606.13003#bib.bib375)\); and \(iii\) code generation throughSWE\-Bench LiteJimenezet al\.\([2024](https://arxiv.org/html/2606.13003#bib.bib437)\)\. However, since these benchmarks prioritize static reasoning, we also includeBrowseComp\-PlusChenet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib66)\)following the recommendation fromKimet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib220)\)to provide a critical test bed for progressive information seeking and dynamic reasoning\. By utilizing these state\-of\-the\-art variants, we aim to reproduce commonly reported MAS improvements and assess their robustness under stringent conditions\.

Automatic MAS Baselines\.We select six representative frameworks that span the current state\-of\-the\-art in autonomous agent coordination, including both inference\-time and optimized \(training/validation based\) variants \(see Appendix[B](https://arxiv.org/html/2606.13003#A2)for complete experimental setup configuration details\):

- •DyLAN\(Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105)\)iteratively selects top\-KKspecialized agents via LLM\-ranking, using dynamic interaction layers to refine team composition from a diverse pool of roles\.
- •MAS\-Zero\(Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207)\)is a zero\-shot framework where a meta\-agent iteratively optimizes multi\-agent orchestrations by selecting from four reasoning blocks \(CoT, CoT\-SC, Reflexion, and Debate\)\. A verifier then evaluates all generated candidate trajectories to select the final response\.
- •ADAS\(Huet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib177)\)employs a meta\-agent to iteratively discover agentic architectures by generating novel coordination code\. Performance metrics from these implementations are stored in an archive to guide subsequent discovery iterations via validation data\.
- •AFlow\(Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589)\)treats workflow design as code\-based search, utilizing Monte Carlo Tree Search \(MCTS\) with an LLM\-based optimizer to iteratively refine candidates based on validation feedback\.
- •MaAS\(Zhanget al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib602)\)uses a controller to sample query\-dependent workflows from a probabilistic supernet, sequentially activating operators until a threshold is met\. This architecture facilitates dynamic early exits and is optimized via textual gradients from environmental feedback\.
- •MAS\-Orchestra\(Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\)employs an RL\-trained orchestrator to manage sub\-agent delegation\. System complexity is governed by the Degree of MAS \(DoM\), where the orchestrator selects sub\-agent configurations \(e\.g\., CoT, Debate\) from a fixed candidate pool based on task requirements\.

Backbone LLMs\.To ensure generalization across paradigms, we evaluate with a stratified selection of LLMs:GPT\-4o,GPT\-5,GPT\-OSS \(120B\), andGemini\-2\.5\-Pro\. This ensemble spans frontier closed\-source models, varied generations, and open\-source alternatives\. While resource and API cost considerations necessitated a focused set of backbone models, this cross\-section allows us to determine if architectural gaps are systemic across different model families and scales\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/plots/plot_ds_rows.png)Figure 2:The MAS Efficiency Frontier\.Cost vs\. accuracy trade\-offs\.CoT\-SCprovides the optimal balance of performance and cost\-efficiency\. Automated MAS \(e\.g\.,ADAS,MAS\-Orchestra\) frequently incur10×10\\timesinference costs vs\. SAS baselines for negligible gains, except onHLE\-Math\. This suggests MAS fails to elevate weaker backbones\. Note: GPT\-OSS\-120B was excluded fromSWE\-Bench Litedue to consistent formatting failures in code patches\.Evaluation Protocol\.Results are averaged across 3 independent runs\.222Gemini\-2\.5\-Pro results use a single run due to cost\.CoT\-SC baseline employs a 5\-sample majority vote across all datasets and backbones\. Appendix[A](https://arxiv.org/html/2606.13003#A1)details test splits\.

### 3\.2Results

Figure[2](https://arxiv.org/html/2606.13003#S3.F2)visualizes the cost\-benefit profile of MAS performance relative to inference expenditure \(including search and validation overhead; see Table[4](https://arxiv.org/html/2606.13003#A2.T4)for the full results\)\. A dominant trend emerges across all benchmarks: CoT\-SC consistently outperforms automated MAS frameworks, frequently achieving higher accuracy at less than10%10\\%of the computational cost\. This suggests that for current frameworks, architectural complexity is an inefficient substitute for simple stochastic sampling\.

Challenging Capability Bridging\.Our results directly challenge the prevailing assumption that sophisticated orchestration can elevate weaker models to frontier\-level performance\(Liet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib250)\): \(i\)No Gains for Mid\-Tier Models:MAS fails to provide consistent improvements for models like GPT\-4o or GPT\-OSS; \(ii\)Model Tier Superiority:A single\-agent GPT\-5 instance using CoT\-SC reliably outperforms the most sophisticated GPT\-4o\-based MAS frameworks \(e\.g\., ADAS or AFlow\) while consuming less than half the total tokens\. These findings indicate that automated MAS designs cannot bridge the generational gap between model tiers; instead, they introduce significant computational bloat without commensurate gains\.

Complexity Requires Competence\.Interestingly, significant MAS uplift only occurs on HLE\-Math using GPT\-5 and Gemini\-2\.5\-Pro\. This suggests a competency floor for MAS: architectural complexity may only yield benefits when the underlying backbone already possesses the high inherent reasoning capabilities necessary to navigate complex coordination\.

Takeaways\.Overall, these findings provide empirical evidence for architectural bloat across the MAS ecosystem\. The significant performance\-cost gap suggests that the sophisticated multi\-agent graphs generated by these frameworks do not translate into functional reasoning gains\. Instead, they represent a failure of automated search to find configurations that outperform unstructured scaling, confirming that current MAS designs have yet to move beyond redundant high\-cost iterations\.

### 3\.3TheSMFRDiagnostic Benchmark

Results from Section[3\.2](https://arxiv.org/html/2606.13003#S3.SS2)show that CoT\-SC outperforms MAS across all standard benchmarking datasets in both accuracy and cost\-effectiveness\. However, existing works such asKapooret al\.\([2024](https://arxiv.org/html/2606.13003#bib.bib201)\); Kimet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib220)\)have critiqued the use of benchmarks created under the assumption of simple input\-output flows for testing MAS\. To isolate task suitability as a factor for the poor performance of MAS, we create a task tailored for multi\-agent workflows called theSynthetic Multi\-Hop Financial Reasoning \(SMFR\)dataset\.

Task Structure\.Each problem presents an agent with a stock price haystack \- historical open/close prices forBBcompanies over a 30\-day window \- and a set of investor transactions \(buy/sell pairs\)\. The agent must determine on which dates each investor could achieve a specified profit or loss target, then identify the winning investor according to an aggregation criterion \(earliest or latest qualifying date\)\. The task is designed to resist shortcut strategies: correct answers require multi\-step context extraction \(price lookup, date lookup, date filtering\) and numerical reasoning \(P&L computation, target price derivation, sorting\)\. Figure[8](https://arxiv.org/html/2606.13003#A3.F8)shows an example instance\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/x1.png)Figure 3:SMFR Dataset Generation Pipeline\.Stock data fromAroussi \([2024](https://arxiv.org/html/2606.13003#bib.bib559)\)is sampled along with parameters such as transaction type, price type, number of investors, etc\. Price tables with distractor data are used to create a haystack; specific transaction prices and dates for investors are the needles that need to be retrieved\. The P&L calculations and winning investor \(answer\) is programmatically computed\.Non\-Linear Interdependence\.Zhuet al\.\([2025](https://arxiv.org/html/2606.13003#bib.bib619)\)establish guidelines for creating agentic benchmarks, which include requirements such as sequential interdependence, where later actions must depend on earlier observations\. AnthropicAnthropic \([2026a](https://arxiv.org/html/2606.13003#bib.bib17)\)recommends that MAS are suitable for tasks where context separation, task parallelization and specialization provide clear benefits\. Following their recommendations,SMFRis explicitly designed to benon\-linearandcontext\-heavy\. Unlike standard QA or mathematical tasks,SMFRcannot be solved via greedy local reasoning\. It requires maintaining a global objective \(target profit\) while executing independent, modular sub\-tasks \(investor\-specific P&L\), including retrieval of information from a large context of historical market data\. A correct solution requires \(i\)Constraint Parsing\(defining targets and comparison logic\); \(ii\)Transaction Extraction\(parsing haystack positions\); \(iii\)P&L Derivation\(establishing realized baselines\); \(iv\)Reverse\-Price Calculation\(deriving required target prices\); \(v\)Threshold Scanning\(validating dates\); and \(vi\)Cross\-Investor Synthesis\(aggregating and selecting the final answer\)\.

Figure[3](https://arxiv.org/html/2606.13003#S3.F3)details the task generation pipeline\.Transaction Extraction,Portfolio P&L Derivation, andReverse\-Price Calculationprovide explicit opportunities for paralellization across investors, while being sequentially dependent within each investor’s trajectory\.

Synthetic Data Generation\.We programmatically generate problems using historical US equity prices\(Aroussi,[2024](https://arxiv.org/html/2606.13003#bib.bib559)\)\. Each instance follows a “Needle\-in\-a\-Haystack” architecture \(Figure[3](https://arxiv.org/html/2606.13003#S3.F3)\): the “Haystack” comprises 30\-day price tables forBBstocks; the task requires models to retrieve specific investor histories \(the “Needles”\) and compute the exact date a target profit/loss threshold was achieved for an open position\. As the specific problem instances are procedurally generated, the benchmark remains immune to data contamination while maintaining the realistic price distributions essential for robust evaluation\. The dataset consists of588588test samples \(\+16\+16for validation\) balanced across transaction types, aggregation logic, and target percentages \(more details and statistics in Appendix[C](https://arxiv.org/html/2606.13003#A3)\)\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/x2.png)Figure 4:Expert\-MAS Pipeline Architecture\.A deterministic, code\-driven architecture serving as competitive baseline\. The system enforces separation of concerns: \(1\) Meta\-Agent parses task topology, \(2\) ExtractorAgent retrieves targeted data, and \(3\) CalculatorAgent reasons over isolated snippets\. A Python orchestrator dispatches these chains concurrently per investor, with final comparisons computed deterministically to ensure high\-precision, low\-noise consistency\.Expert\-Designed MAS\.To establish a competitive reference baseline for performance onSMFR, we design anExpert\-MASbased on guidelines from AnthropicAnthropic \([2026a](https://arxiv.org/html/2606.13003#bib.bib17)\)that utilizes structured decomposition and deterministic orchestration\. Expert\-MAS enforces a strict separation between context processing and logical control \(Figure[4](https://arxiv.org/html/2606.13003#S3.F4)\), decomposing the task into a multi\-step pipeline where a Meta\-Agent first parses the problem topology into a structured schema\. This schema then drives a deterministic Python\-based Executor that orchestrates specialized sub\-agents for targeted retrieval and numerical reasoning\. By offloading task coordination and final win\-determination to deterministic code, Expert\-MAS minimizes context bloat and eliminates the “orchestration noise” prevalent in automated MAS designs\. Appendix[D](https://arxiv.org/html/2606.13003#A4)details the full configuration setup\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/plots/plot_stocks_v6.png)Figure 5:Automated MAS consistently fail to surpassCoT\-SCefficiency onSMFRas well\. Expert\-MAS achieves superior trade\-offs except on GPT\-4o \(bottlenecked by base\-model reasoning limits\)\. Gemini\-2\.5\-Pro is omitted due to non\-viable MAS cost multipliers \(\>10×\>10\\times\)\.Results\. Our benchmark serves as an agentic stress test: GPT\-5 reaches only57\.0%57\.0\\%accuracy with CoT\-SC, while GPT\-4o and GPT\-OSS struggle between22\.1%22\.1\\%and26\.1%26\.1\\%\(Table[4](https://arxiv.org/html/2606.13003#A2.T4)\)\. Despite explicit agentic requirements \(multi\-step planning, state tracking, long\-context retrieval\), automated MAS frameworks rarely surpass CoT\-SC and never do so economically \(Figure[5](https://arxiv.org/html/2606.13003#S3.F5)\)\. The three statistically significant improvements \- DyLAN on GPT\-OSS \(\+6\.6\+6\.6pp,5×5\\timescost\), DyLAN on GPT\-5 \(\+4\.3\+4\.3pp,2\.5×2\.5\\timescost\), and MAS\-Orchestra on GPT\-5 \(\+6\.0\+6\.0pp,1\.9×1\.9\\timescost\) \- occur exclusively on stronger backbones and at substantial overhead, while GPT\-4o yields no significant gains from any automated framework\. In contrast, Expert\-MAS achieves substantial performance improvements with cost comparable to CoT\-SC: GPT\-OSS improves from26\.1%26\.1\\%to36\.1%36\.1\\%, while GPT\-5 jumps from57\.0%57\.0\\%to a near\-perfect96\.5%96\.5\\%\.333Full Gemini\-2\.5\-Pro MAS evaluations were excluded as the\>10×\>10\\timescost multipliers \(Section[3](https://arxiv.org/html/2606.13003#S3)\) rendered them non\-viable\.The sole exception is GPT\-4o, where persistent calculation and retrieval failures bottleneck the system regardless of orchestration\. This reinforces our finding that MAS require a threshold baseline competency to be effective\. Thus, while the MAS paradigm is fundamentally viable, current automated frameworks fail to exploit task\-specific opportunities effectively or economically\.

## 4Architectural Deconstruction

While results from Section[3](https://arxiv.org/html/2606.13003#S3)establish a clear efficiency gap between single\- and multi\-agent systems, they do not reveal whether the internal mechanisms of MAS, such as role specialization and consensus, provide latent benefits that justify their complexity\. To address this, we deconstruct the generated architectures and investigate whether their features contribute meaningfully to the reasoning process\. We find that in most automatically generated workflows, these mechanisms are either sub\-optimal or purely decorative rather than emergent intelligence\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/img/selection_histograms_4datasets.png)Figure 6:Judge model selection frequency of MAS\-Zero across four datasets, using GPT\-4o \(blue\) and GPT\-5 \(orange\) as both worker and verifier\. Indices 0–3 correspond to four fundamental reasoning paradigms: vanilla CoT, CoT\-SC, Reflexion, and Debate\. Indices 4–8 represent the subsequent 5 rounds of multi\-agent organization search\.Functional Collapse and Structural Redundancy\.Frameworks like DyLAN\(Liuet al\.,[2024a](https://arxiv.org/html/2606.13003#bib.bib105)\)posit that performance is driven by “agent diversity,” yet our analysis reveals this fails to manifest in practice\. Instead, we observe a functional collapse where agents reach immediate, unanimous consensus in∼70%\\sim 70\\%of GPT\-4o cases and\>90%\>90\\%of GPT\-5 cases, effectively functioning as a unanimous CoT\-SC baseline rather than a dynamic negotiation\. In cases where interaction does occur, task\-specific roles provide no marginal utility; an “all\-assistant” configuration achieved better accuracy than task\-specific experts \(54\.4%54\.4\\%vs\.53\.4%53\.4\\%; see Appendix[E\.1](https://arxiv.org/html/2606.13003#A5.SS1)for experiment details\)\. Similarly, in MAS\-Zero\(Keet al\.,[2025b](https://arxiv.org/html/2606.13003#bib.bib207)\), a dedicated verifier aggregates worker outputs to select the optimal result\. However, our analysis across four benchmarks reveals a systematic positional bias that triggers consensus collapse\. Across all tested models, the verifier disproportionately favors earlier entries in the context window: GPT\-4o selects the initial block in over45%45\\%of instances, while GPT\-5 demonstrates a slightly broader but still heavily front\-loaded preference \(see Figure[6](https://arxiv.org/html/2606.13003#S4.F6)and Appendix[E\.3](https://arxiv.org/html/2606.13003#A5.SS3)for selection frequency distributions\)\. Conversely, outputs from later search rounds are rarely selected, accounting for less than15%15\\%of final decisions\. This structural redundancy turns subsequent worker agents into “expensive witnesses” that incur full inference costs while exerting near\-zero causal influence on the output\.

![Refer to caption](https://arxiv.org/html/2606.13003v1/plots/adas_gpqa_gpt5_search_dynamics2.png)Figure 7:ADAS \(GPT\-5\) validation accuracies on different seeded runs on GPQA\-diamond dataset\.Convergence on Heuristic Search Artifacts\.Our analysis suggests that frameworks designed to discover architectures \(ADAS\(Huet al\.,[2024](https://arxiv.org/html/2606.13003#bib.bib174)\), AFlow\(Zhanget al\.,[2025c](https://arxiv.org/html/2606.13003#bib.bib589)\)\) function as heuristic explorers rather than principled optimizers\. On GPQA\-Diamond, ADAS search dynamics are non\-monotonic; accuracy frequently peaks early and subsequently regresses \(Figure[7](https://arxiv.org/html/2606.13003#S4.F7)\), suggesting that performance gains are stochastic “lucky” iterations rather than structural evolution\. This is supported by our motif analysis, where we map generated architectures to a rule\-based dictionary \(e\.g\., Self\-consistency, Aggregation, Verifier\)\. Across all settings, the primary positive signal originated from Self\-consistency motifs\. On GPQA\-Diamond, architectures incorporating these motifs achieved a mean accuracy of82\.19%82\.19\\%\(\+1\.34%\+1\.34\\%over the global average\); specialized coordination motifs yielded negligible gains\. Our inspection of AFlow reveals a similar issue: instead of manifesting complex coordination, the discovered MAS consistently degenerate into trivial ensembles\. As illustrated in our case analysis \(Figure[10](https://arxiv.org/html/2606.13003#A5.F10)\), “optimized” workflows frequently converge on a structure that simply iterates a single custom prompt three times before aggregation \- a configuration functionally identical to standard CoT\-SC\. Across 14 final workflows generated by GPT\-4o, GPT\-5, and GPT\-OSS\-120B on five datasets,50%50\\%\(7/147/14\) adopted this simplistic structure, with four of these actually underperforming the CoT\-SC baseline\. This evidence confirms that automated search often converges on rediscovering CoT\-SC style sampling under more complex labels, rather than inventing novel multi\-agent strategies\.

Table 1:Operator Activation Distribution \(%\) of MaAS \(GPT\-5\)\. On context\-heavy BrowseComp\-Plus, the controller collapses to I/O calls due to cost\-dominated optimization; while on GPQA\-Diamond, it spreads calls more evenly but fails to outperform CoT\-SC\.
Table 2:Agent Selection Distribution \(%\) by MAS\-Orchestra Across Datasets\.

Incentive Misalignment in Dynamic Routing\.In systems designed for adaptive orchestration \(MaAS\(Zhanget al\.,[2025a](https://arxiv.org/html/2606.13003#bib.bib598)\), MAS\-Orchestra\(Keet al\.,[2026](https://arxiv.org/html/2606.13003#bib.bib209)\)\), the optimization objectives often fail to produce meaningful routing logic\. In MaAS, the use of highly capable base models \(e\.g\.,GPT\-5\) flattens the accuracy gradient to∼1/K\\sim 1/K, causing the controller to ignore task\-specific logic and collapse into two distinct failure modes: \(1\)Cost\-Minimizing Collapseon BrowseComp\-Plus, where74\.2%74\.2\\%of activations are a trivial, single I/O call; and \(2\)Stochastic Stallingon GPQA\-Diamond, where negligible cost differentials trap the controller in its initialized near\-uniform distribution \(Table[2](https://arxiv.org/html/2606.13003#S4.T2)\)\. Similarly, MAS\-Orchestra exhibits a difficulty\-agnostic policy\. Across all benchmarks, the system largely ignores its diverse agent pool, converging instead on a rigid binary preference for high\-overhead Debate and Reflexion agents \(Table[2](https://arxiv.org/html/2606.13003#S4.T2)\)\. The orchestrator fails to scale agent complexity to task difficulty; despite GPQA\-Diamond posing a lower reasoning ceiling than HLE\-Math, the system exhibited a higher reliance on Debate agents for the former \(84\.9%84\.9\\%\) than the latter \(79\.2%79\.2\\%\)\. These behaviors confirm that automated orchestrators do not learn task\-adaptive strategies, but instead settle into static, greedy local minima\.

## 5Discussion

Our evaluation reveals a systematic divergence between the theoretical complexity of MAS frameworks and their empirical execution\. While intended to foster emergent collaboration, current automated paradigms frequently result in mechanistic trivialization\.

The Ensembling Trap\.A primary driver of this collapse is the reliance on CoT and CoT\-SC as the fundamental building blocks of MAS\. While using these primitives ensures generalization and leverages ensembling effects, the resulting architectures fail to implement them efficiently\. Instead of synergistic coordination, frameworks like AFlow and ADAS often settle into structural degeneration, rediscovering basic ensembling motifs under the guise of an optimized graph\. The∼10×\\sim 10\\timesincrease in cost thus buys little more than a redundant, poorly routed version of a standard CoT\-SC baseline\.

Towards Mechanistic Interpretability of MAS\.As model capability scales, the MAS advantage further erodes due to two factors: \(i\)Signal Saturation:in models like GPT\-5, accuracy gradients flatten, causing controllers \(MaAS\) to lose the signal needed for nuanced routing, leading to either cheap shortcuts or static policy collapse; \(ii\)Positional and Primacy Biases:verifiers and controllers \(MAS\-Zero, DyLAN\) disproportionately favor early reasoning steps, effectively terminating the multi\-agent benefit before interaction occurs\. The success of Expert\-MAS on theSMFRbenchmark reinforces the findings of\(Anthropic,[2026a](https://arxiv.org/html/2606.13003#bib.bib17)\)and\(Kimet al\.,[2025](https://arxiv.org/html/2606.13003#bib.bib220)\): multi\-agent coordination excels only when architectures are specifically engineered to exploit parallelizable sub\-problems or context protection\. Future research should pivot away from black\-box automated graph generation that tends to default to redundant ensembling, and toward the mechanistic interpretability of agent interactions\. We argue that to move beyond creating “expensive witnesses,” MAS must be evaluated on their structural fidelity: the degree to which assigned agentic roles exert measurable causal influence on the final decision\. Without such grounding, increased architectural complexity serves only to mask computational inefficiency\.

## 6Conclusion

Our systematic evaluation identifies a critical efficiency gap in modern MAS design, where architectural complexity often masks a fundamental functional collapse into simpler, stochastic baselines\. By introducing theSMFRbenchmark and isolating the mechanistic failures of six major frameworks, we provide a roadmap for more principled, cost\-effective agentic design\. Our architectural deconstruction reveals that current automated workflows frequently degenerate into redundant ensembling loops functionally identical to CoT\-SC\. Ultimately, our findings suggest that moving beyond “expensive witnesses” requires a pivot from black\-box graph searching towards architectures grounded in verifiable task decomposition and causal role\-alignment\.

## References

- How we built our multi\-agent research system\.Note:[https://www\.anthropic\.com/engineering/built\-multi\-agent\-research\-system](https://www.anthropic.com/engineering/built-multi-agent-research-system)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- Anthropic \(2026a\)Building multi\-agent systems: when and how to use them\.Note:[hhttps://claude\.com/blog/building\-multi\-agent\-systems\-when\-and\-how\-to\-use\-them](https://arxiv.org/html/2606.13003v1/hhttps://claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p4.1),[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p3.1),[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p6.1),[§5](https://arxiv.org/html/2606.13003#S5.p3.1)\.
- Anthropic \(2026b\)Claude code agent teams\.Note:[https://code\.claude\.com/docs/en/agent\-teams](https://code.claude.com/docs/en/agent-teams)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p2.1)\.
- R\. Aroussi \(2024\)Yfinance: yahoo\! finance market data downloader\.GitHub\.Note:[https://github\.com/ranaroussi/yfinance](https://github.com/ranaroussi/yfinance)Cited by:[Appendix C](https://arxiv.org/html/2606.13003#A3.p2.1),[Figure 3](https://arxiv.org/html/2606.13003#S3.F3),[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p5.3)\.
- M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica \(2025\)Why do multi\-agent llm systems fail?\.External Links:2503\.13657,[Link](https://arxiv.org/abs/2503.13657)Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p5.1)\.
- W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\. Qian, C\. Chan, Y\. Qin, Y\. Lu, R\. Xie, Z\. Liu, M\. Sun, and J\. Zhou \(2023\)AgentVerse: facilitating multi\-agent collaboration and exploring emergent behaviors in agents\.ArXivabs/2308\.10848\.External Links:[Link](https://api.semanticscholar.org/CorpusID:261048935)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su, S\. Sharifymoghaddam, Y\. Li, H\. Hong, X\. Shi, X\. Liu, N\. Thakur, C\. Zhang, L\. Gao, W\. Chen, and J\. Lin \(2025\)BrowseComp\-plus: a more fair and transparent evaluation benchmark of deep\-research agent\.arXiv preprint arXiv:2508\.06600\.Cited by:[4th item](https://arxiv.org/html/2606.13003#A1.I1.i4.p1.1.1),[§1](https://arxiv.org/html/2606.13003#S1.p1.1),[§1](https://arxiv.org/html/2606.13003#S1.p4.1),[§3\.1](https://arxiv.org/html/2606.13003#S3.SS1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p4.1)\.
- J\. N\. Foerster, Y\. Assael, N\. de Freitas, and S\. Whiteson \(2016\)Learning to communicate with deep multi\-agent reinforcement learning\.ArXivabs/1605\.06676\.External Links:[Link](https://api.semanticscholar.org/CorpusID:53391180)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1)\.
- M\. Gao, Y\. Li, B\. Liu, Y\. Yu, P\. Wang, C\. Lin, and F\. Lai \(2025\)Single\-agent or multi\-agent systems? why not both?\.ArXivabs/2505\.18286\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278904492)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p4.1)\.
- P\. Hernandez\-Leal, B\. Kartal, and M\. E\. Taylor \(2018\)A survey and critique of multiagent deep reinforcement learning\.Autonomous Agents and Multi\-Agent Systems33,pp\. 750 – 797\.External Links:[Link](https://api.semanticscholar.org/CorpusID:202540003)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1)\.
- S\. Hu, C\. Lu, and J\. Clune \(2024\)Automated design of agentic systems\.arXiv preprint arXiv:2408\.08435\.Cited by:[§E\.4](https://arxiv.org/html/2606.13003#A5.SS4),[§E\.4](https://arxiv.org/html/2606.13003#A5.SS4.SSS0.Px1.p1.1),[§E\.4](https://arxiv.org/html/2606.13003#A5.SS4.p1.1),[§1](https://arxiv.org/html/2606.13003#S1.p2.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§4](https://arxiv.org/html/2606.13003#S4.p3.4)\.
- S\. Hu, C\. Lu, and J\. Clune \(2025\)Automated design of agentic systems\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=t9U3LW7JVX)Cited by:[3rd item](https://arxiv.org/html/2606.13003#A2.I1.i3.p1.1),[§2](https://arxiv.org/html/2606.13003#S2.p3.1),[3rd item](https://arxiv.org/html/2606.13003#S3.I1.i3.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[3rd item](https://arxiv.org/html/2606.13003#A1.I1.i3.p1.1.1),[§3\.1](https://arxiv.org/html/2606.13003#S3.SS1.p1.1)\.
- S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. Narayanan \(2024\)Ai agents that matter\.arXiv preprint arXiv:2407\.01502\.External Links:[Link](https://arxiv.org/abs/2407.01502)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§2](https://arxiv.org/html/2606.13003#S2.p1.1),[§2](https://arxiv.org/html/2606.13003#S2.p5.1),[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p1.1)\.
- Z\. Ke, F\. Jiao, Y\. Ming, X\. Nguyen, A\. Xu, D\. X\. Long, M\. Li, C\. Qin, P\. Wang, S\. Savarese, C\. Xiong, and S\. Joty \(2025a\)A survey of frontiers in llm reasoning: inference scaling, learning to reason, and agentic systems\.TMLR\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1)\.
- Z\. Ke, Y\. Ming, A\. Xu, R\. Chin, X\. Nguyen, P\. Jwalapuram, J\. Wang, S\. Yavuz, C\. Xiong, and S\. Joty \(2026\)Mas\-orchestra: understanding and improving multi\-agent reasoning through holistic orchestration and controlled benchmarks\.ICML\.Cited by:[6th item](https://arxiv.org/html/2606.13003#A2.I1.i6.p1.1),[§E\.6](https://arxiv.org/html/2606.13003#A5.SS6),[§E\.6](https://arxiv.org/html/2606.13003#A5.SS6.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.13003#S1.p2.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§1](https://arxiv.org/html/2606.13003#S1.p4.1),[§2](https://arxiv.org/html/2606.13003#S2.p3.1),[6th item](https://arxiv.org/html/2606.13003#S3.I1.i6.p1.1),[§4](https://arxiv.org/html/2606.13003#S4.p4.4)\.
- Z\. Ke, A\. Xu, Y\. Ming, X\. Nguyen, C\. Xiong, and S\. Joty \(2025b\)MAS\-ZERO: designing multi\-agent systems with zero supervision\.SEA@NeurIPS\.Cited by:[2nd item](https://arxiv.org/html/2606.13003#A2.I1.i2.p1.1),[§E\.3](https://arxiv.org/html/2606.13003#A5.SS3),[§E\.3](https://arxiv.org/html/2606.13003#A5.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.13003#S1.p2.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§2](https://arxiv.org/html/2606.13003#S2.p2.1),[2nd item](https://arxiv.org/html/2606.13003#S3.I1.i2.p1.1),[§4](https://arxiv.org/html/2606.13003#S4.p2.6)\.
- Y\. H\. Kim, K\. Gu, C\. Park, C\. Park, S\. Schmidgall, A\. A\. Heydari, Y\. Yan, Z\. Zhang, Y\. Zhuang, Y\. Liu, M\. Malhotra, P\. P\. Liang, H\. W\. Park, Y\. Yang, X\. Xu, Y\. Du, S\. N\. Patel, T\. Althoff, D\. McDuff, and X\. Liu \(2025\)Towards a science of scaling agent systems\.ArXivabs/2512\.08296\.External Links:[Link](https://api.semanticscholar.org/CorpusID:283712193)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§1](https://arxiv.org/html/2606.13003#S1.p4.1),[§2](https://arxiv.org/html/2606.13003#S2.p5.1),[§3\.1](https://arxiv.org/html/2606.13003#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p1.1),[§5](https://arxiv.org/html/2606.13003#S5.p3.1)\.
- J\. Li, Q\. Zhang, Y\. Yu, Q\. Fu, and D\. Ye \(2024\)More agents is all you need\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=bgzUSZ8aeg)Cited by:[§3\.2](https://arxiv.org/html/2606.13003#S3.SS2.p2.1)\.
- Z\. Liu, Y\. Zhang, P\. Li, Y\. Liu, and D\. Yang \(2024a\)A dynamic llm\-powered agent network for task\-oriented agent collaboration\.arXiv preprint arXiv:2310\.02170\.Cited by:[1st item](https://arxiv.org/html/2606.13003#A2.I1.i1.p1.1),[§E\.1](https://arxiv.org/html/2606.13003#A5.SS1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§2](https://arxiv.org/html/2606.13003#S2.p2.1),[1st item](https://arxiv.org/html/2606.13003#S3.I1.i1.p1.1),[§4](https://arxiv.org/html/2606.13003#S4.p2.6)\.
- Z\. Liu, Y\. Zhang, P\. Li, Y\. Liu, and D\. Yang \(2024b\)A dynamic llm\-powered agent network for task\-oriented agent collaboration\.arXiv preprint arXiv:2310\.02170\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p2.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=S37hOerQLB)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2024\)Self\-refine: iterative refinement with self\-feedback\.Advances in Neural Information Processing Systems36\.Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p1.1)\.
- MiroFish \(2026\)MiroFish: a simple and universal swarm intelligence engine\.Note:[https://github\.com/666ghj/MiroFish](https://github.com/666ghj/MiroFish)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p2.1)\.
- OpenClaw Agents \(2026\)OpenClaw agents: a multi\-agent configuration kit for openclaw\.Note:[https://github\.com/shenhao\-stu/openclaw\-agents](https://github.com/shenhao-stu/openclaw-agents)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p2.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[2nd item](https://arxiv.org/html/2606.13003#A1.I1.i2.p1.1.1),[§1](https://arxiv.org/html/2606.13003#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.13003#S3.SS1.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)GPQA: a graduate\-level google\-proof qa benchmark\.External Links:2311\.12022,[Link](https://arxiv.org/abs/2311.12022)Cited by:[1st item](https://arxiv.org/html/2606.13003#A1.I1.i1.p1.1.1),[§3\.1](https://arxiv.org/html/2606.13003#S3.SS1.p1.1)\.
- H\. Su, S\. Diao, X\. Lu, M\. Liu, J\. Xu, X\. Dong, Y\. Fu, P\. Belcak, H\. Ye, H\. Yin, Y\. Dong, E\. Bakhturina, T\. Yu, Y\. Choi, J\. Kautz, and P\. Molchanov \(2025\)ToolOrchestra: elevating intelligence via efficient model and tool orchestration\.External Links:2511\.21689,[Link](https://arxiv.org/abs/2511.21689)Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p3.1)\.
- D\. Tran and D\. Kiela \(2026\)Single\-agent llms outperform multi\-agent systems on multi\-hop reasoning under equal thinking token budgets\.External Links:2604\.02460,[Link](https://arxiv.org/abs/2604.02460)Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p5.1)\.
- V\. Venkataramani, H\. Shi, Z\. Ke, A\. Xu, X\. He, Y\. Zhou, S\. Yavuz, H\. Wang, and S\. Joty \(2026\)MAS\-prove: understanding the process verification of multi\-agent systems\.ICML\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- J\. Wang, J\. Wang, B\. Athiwaratkun, C\. Zhang, and J\. Zou \(2024\)Mixture\-of\-agents enhances large language model capabilities\.ArXivabs/2406\.04692\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270357878)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1)\.
- W\. Wang, J\. Hao, Y\. Wang, and M\. E\. Taylor \(2019\)Achieving cooperation through deep multiagent reinforcement learning in sequential prisoner’s dilemmas\.Proceedings of the First International Conference on Distributed Artificial Intelligence\.External Links:[Link](https://api.semanticscholar.org/CorpusID:53360551)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- A\. Wynn, H\. Satija, and G\. K\. Hadfield \(2025\)Talk isn’t always cheap: understanding failure modes in multi\-agent debate\.ArXivabs/2509\.05396\.External Links:[Link](https://api.semanticscholar.org/CorpusID:281203300)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou, R\. Zheng, X\. Fan, X\. Wang, L\. Xiong, Y\. Zhou, W\. Wang, C\. Jiang, Y\. Zou, X\. Liu, Z\. Yin, S\. Dou, R\. Weng, W\. Cheng, Q\. Zhang, W\. Qin, Y\. Zheng, X\. Qiu, X\. Huang, and T\. Gui \(2023\)The rise and potential of large language model based agents: a survey\.arXiv preprint arXiv:2309\.07864\.Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§2](https://arxiv.org/html/2606.13003#S2.p1.1)\.
- G\. Zhang, L\. Niu, J\. Fang, K\. Wang, L\. Bai, and X\. Wang \(2025a\)Multi\-agent architecture search via agentic supernet\.arXiv preprint arXiv:2502\.04180\.Cited by:[§E\.5](https://arxiv.org/html/2606.13003#A5.SS5),[§E\.5](https://arxiv.org/html/2606.13003#A5.SS5.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.13003#S1.p2.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§2](https://arxiv.org/html/2606.13003#S2.p3.1),[§4](https://arxiv.org/html/2606.13003#S4.p4.4)\.
- G\. Zhang, L\. Niu, J\. Fang, K\. Wang, L\. Bai, and X\. Wang \(2025b\)Multi\-agent architecture search via agentic supernet\.External Links:2502\.04180,[Link](https://arxiv.org/abs/2502.04180)Cited by:[5th item](https://arxiv.org/html/2606.13003#A2.I1.i5.p1.3),[5th item](https://arxiv.org/html/2606.13003#S3.I1.i5.p1.1)\.
- J\. Zhang, J\. Xiang, Z\. Yu, F\. Teng, X\. Chen, J\. Chen, M\. Zhuge, X\. Cheng, S\. Hong, J\. Wang, B\. Zheng, B\. Liu, Y\. Luo, and C\. Wu \(2025c\)AFlow: automating agentic workflow generation\.External Links:[Link](https://openreview.net/forum?id=z5uVAKwmjf)Cited by:[4th item](https://arxiv.org/html/2606.13003#A2.I1.i4.p1.1),[§E\.2](https://arxiv.org/html/2606.13003#A5.SS2),[§E\.2](https://arxiv.org/html/2606.13003#A5.SS2.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.13003#S1.p2.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1),[§2](https://arxiv.org/html/2606.13003#S2.p3.1),[4th item](https://arxiv.org/html/2606.13003#S3.I1.i4.p1.1),[§4](https://arxiv.org/html/2606.13003#S4.p3.4)\.
- Y\. Zhang, X\. Liu, and C\. Xiao \(2025d\)MetaAgent: automatically constructing multi\-agent systems based on finite state machines\.ArXivabs/2507\.22606\.External Links:[Link](https://api.semanticscholar.org/CorpusID:280391843)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- H\. Zhou, X\. Wan, X\. Wan, R\. Sun, H\. Palangi, S\. Iqbal, I\. Vuli’c, A\. Korhonen, and S\. Ö\. Arik \(2025\)Multi\-agent design: optimizing agents with better prompts and topologies\.ArXivabs/2502\.02533\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276107353)Cited by:[§1](https://arxiv.org/html/2606.13003#S1.p1.1),[§1](https://arxiv.org/html/2606.13003#S1.p3.1)\.
- Y\. Zhu, T\. Jin, Y\. Pruksachatkun, A\. K\. Zhang, S\. Liu, S\. Cui, S\. Kapoor, S\. Longpre, K\. Meng, R\. Weiss, F\. Barez, R\. Gupta, J\. Dhamala, J\. Merizian, M\. Giulianelli, H\. Coppock, C\. Ududec, A\. Kellermann, J\. S\. Sekhon, J\. Steinhardt, S\. Schwettmann, A\. Narayanan, M\. Zaharia, I\. Stoica, P\. Liang, and D\. Kang \(2025\)Establishing best practices in building rigorous agentic benchmarks\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=E58HNCqoaA)Cited by:[§3\.3](https://arxiv.org/html/2606.13003#S3.SS3.p3.1)\.

## Appendix ABenchmark Dataset Details

The benchmark datasets across mathematical reasoning, QA and coding used in the paper are described below\. See Table[3](https://arxiv.org/html/2606.13003#A1.T3)for the validation and test samples splits used in our experiments\.

- •GPQA Diamond\[[29](https://arxiv.org/html/2606.13003#bib.bib375)\]: is a high\-difficulty, multiple\-choice science benchmark comprising 198 questions across biology, physics, and chemistry\. Unlike other subsets of GPQA, the “Diamond” set is restricted to questions where highly educated subject\-matter experts \(SMEs\) agree on the correct answer, yet non\-expert humans—even when equipped with unrestricted web access—fail to answer correctly\. This makes GPQA Diamond a rigorous test of expert\-level reasoning and a benchmark for evaluating whether LLMs can transcend general\-purpose knowledge\.
- •HLE\-Maths\[[28](https://arxiv.org/html/2606.13003#bib.bib340)\]: is a subset ofHumanity’s Last Exam \(HLE\), a benchmark composed of graduate\-level problems across specialized mathematical fields\. The dataset is explicitly designed to be closed\-source to prevent data contamination and consists of questions that are non\-trivial for subject\-matter experts\. Unlike earlier benchmarks like MATH or GSM8K, HLE Maths focuses on multi\-step abstract reasoning and complex theorem application where the search space for a correct solution is vast\. It serves as a high\-resolution probe for whether LLMs \- and by extension, MAS architectures \- can navigate the extreme reasoning depth required for original mathematical research\.
- •SWE\-Bench Lite\[[15](https://arxiv.org/html/2606.13003#bib.bib437)\]: is a curated subset of tasks from the full SWE\-bench dataset, designed to evaluate an agent’s ability to resolve real\-world GitHub issues within popular open\-source Python repositories \(e\.g\., django, scikit\-learn, sympy\)\. Unlike synthetic coding benchmarks, it requires end\-to\-end agentic behavior: the model must navigate a sprawling file system, localize the bug across multiple modules, and generate a precise \.patch file that passes a hidden suite of unit tests\. Its “Lite” designation ensures the tasks are self\-contained enough for evaluation while maintaining the high\-dimensional context and multi\-step planning required for professional software maintenance\.
- •BrowseComp\-Plus\[[7](https://arxiv.org/html/2606.13003#bib.bib66)\]: is a fair and transparent benchmark for deep\-research agents, derived from the original BrowseComp\. Unlike its predecessor, which relies on dynamic and opaque web search APIs, BrowseComp\-Plus employs a fixed, human\-verified corpus of over 100,000 documents\. This static environment allows for the disentanglement of retrieval and reasoning components, enabling researchers to isolate whether an agent’s failure is due to poor search query formulation or an inability to synthesize evidence from multiple sources\. Each of its questions is designed to be deep\-research in nature, requiring iterative sub\-problem decomposition and the synthesis of information across diverse web documents\.

Table 3:Data size for different splits in each dataset\.
## Appendix BAutomatic MAS Baseline Configuration Details

The complete experimental setup and configuration details used for the evaluations in our work are described below\. In terms of the LLM parameter settings, we adopt the defaulttemperaturevalues specific to each automatic\-MAS,max\_tokens=32K, andreasoning\_effort=medium for reasoning LLMs\. The full experiment results and costs can be found in Table[4](https://arxiv.org/html/2606.13003#A2.T4)\.

- •DyLAN\[[22](https://arxiv.org/html/2606.13003#bib.bib105)\]: We follow the default settings that use four agents for the team,K=2K=2, and a maximum of three rounds\. Specifically, the four agents are configured with a general “Assistant” role alongside three domain\-specific expert roles tailored to each dataset’s context\. Following their practice, we leverage an LLM \(GPT\-5\) to generate these expert roles and adopt aTheoretical Physicist,Molecular Chemist, andCellular Biologistfor GPQA Diamond; aMathematician,Algebra Expert, andGeometry Wizardfor HLE\-Maths; aProgrammer,Code Reviewer, andSoftware Engineerfor SWE\-Bench Lite; aKnowledge Researcher,Cultural Historian, andInformation Analystfor BrowseComp\-Plus; and aFinancial Analyst,Data Scientist, andProgrammerfor SMFR\. Full system prompt for these expert roles can be found in Table[5](https://arxiv.org/html/2606.13003#A2.T5)\.
- •MAS\-Zero\[[19](https://arxiv.org/html/2606.13003#bib.bib207)\]: We adhere to the original search pipeline consisting of four fundamental blocks followed by subsequent iterations of meta\-agent orchestration\. The verifier utilizes the same backbone model as the meta\-agent\. The system defaults to vanilla CoT output in case of verifier failure\. Defaulting to CoT only occurs in cases of technical exceptions, such as context window overflow\.
- •ADAS\[[14](https://arxiv.org/html/2606.13003#bib.bib177)\]: Unlike the original work, which used varied LLMs to reduce costs, we standardize the backbone to maintain architectural parity\. We maintain the default search depth of 30 iterations to evaluate the trajectory of architectural evolution with some exceptions\. Primarily when using GPT\-5 as the backbone, we reduce the number of iterations due to high inference time and cost\. Specifically: BCP with GPT\-5 uses 10 iterations; HLE\-math with GPT\-5 uses 15; SWE\-bench uses 10 iterations with GPT\-4o and 5 with GPT\-5; and SMFR uses 15 iterations with GPT\-4o and 10 with GPT\-5\. We set the maximum number of debugging attempts to 3, following the original ADAS implementation\.
- •AFlow\[[42](https://arxiv.org/html/2606.13003#bib.bib589)\]: Following the original protocol, we use the default 20\-round search budget, with each candidate workflow being evaluated five times during the search stage to provide stable performance feedback for the MCTS optimizer\. We adopt Custom \(I/O\), AnswerGenerate \(CoT\), and ScEnsemble \(Aggregation\) operators for all the datasets\.
- •MaAS\[[41](https://arxiv.org/html/2606.13003#bib.bib602)\]: We follow default settings to retrain the supernet on each benchmark to evaluate its ability to learn task\-specific routing across diverse reasoning and coding operators\. We use the default hyperparameters: sampling countK=4K\{=\}4, a maximum ofL=4L\{=\}4layers, and an activation threshold of0\.30\.3and one round of training\. The operator pool includes I/O \(direct answer generation\), CoT \(single chain\-of\-thought\), CoT\-SC \(multiple chain\-of\-thought\), ScEnsemble \(majority voting over candidates\), SelfRefine \(critique\-and\-revise\), and EarlyStop \(early exit\), with Programmer additionally for HLE\-Maths\.
- •MAS\-Orchestra\[[18](https://arxiv.org/html/2606.13003#bib.bib209)\]: The candidate pool consists of four fixed sub\-agents: CoT, CoT\-SC, Reflexion, and Debate\. A key design parameter is the “Degree of MAS” \(DoM\), capturing the degree of multi\-agent coordination appropriate for a given task: under low DoM, the orchestrator decides whether to delegate tasks and how to configure the selected sub\-agents, while high DoM additionally requires determining the inter\-agent topology\. In our evaluation, we follow the default setting that uses the officially released orchestrator model with DoM=Low to analyze how effectively it routes queries to specialized reasoning architectures\.

Table 4:Accuracy \(%\) and cost \($\) for all systems across datasets and LLMs\. Dashes indicate missing runs\. CoT / CoT\-SC report the best observed score across MAS system entries\. Expert MAS \(SMFR only\) is a human\-designed multi\-agent system evaluated on the SMFR benchmark\. Note that GPT\-OSS\-120B is not evaluated on SWE\-Bench Lite due as it fails to consistently generate code patches in the required format\. We also exclude MAS\-Orchestra experiments and report single run results for GPT\-5/Gemini\-2\.5\-Pro for other datasets due to significant cost multipliers\.Table 5:Role configurations and corresponding system prompts for each dataset in DyLAN\.
## Appendix CSynthetic Data Generation Details

![Refer to caption](https://arxiv.org/html/2606.13003v1/img/plot_dataset_sample.png)Figure 8:Sample instance ofSMFRtask with 3 investorsTable 6:SMFR dataset statistics perNN\(number of investors \- parallelizable\)\. Samples exclude the held\-out validation set \(16 samples\)\. Avg prompt tokens estimated from problem text \(words×\\times1\.3\)\.NNSamplesNo\-winner \(%\)Avg stocks in contextAvg tx monthsAvg prompt tokens22967%4\.05\.21,88333964%6\.015\.02,9374410410%8\.018\.13,9935515211%10\.019\.05,399661409%10\.019\.05,846Total5889%8\.015\.94,281

The dataset is balanced across six axes: question type∈\\insell, buy, aggregation∈\\inearliest, latest, price type∈\\inopen, close, 13 target percentages from 0\.1% to 2\.0%, and uniformly sampled investor counts and distractor counts in the range\[2,6\]\[2,6\]\. As a synthetic dataset, it avoids issues with model contamination; it is also designed to be updateable to the latest stock prices without having to regenerate the entire sample set\. Sample instance is shown in Figure[8](https://arxiv.org/html/2606.13003#A3.F8), and dataset statistics in Table[6](https://arxiv.org/html/2606.13003#A3.T6)\.

Problems are generated programmatically using real historical stock prices fetched via\[[4](https://arxiv.org/html/2606.13003#bib.bib559)\]for US equities \(e\.g\. AAPL, MSFT, GOOG, etc\.\), ensuring that numerical reasoning is performed on realistic distributions rather than uniform random noise\. The full dataset generation pipeline from Figure[3](https://arxiv.org/html/2606.13003#S3.F3)is detailed below:

1. 1\.Stock Data Sampling\.For each sample, we randomly select a target transaction type \(buy/sell\), price type \(open/close\), and a target profit/loss percentage\. The number of investors \(parallelizable threads\), the breadthBB\(total number of stocks traded\), and the depthDD\(number of transactions per investor\) of the dataset are varied to give us a range of context sizes and task difficulty\.
2. 2\.Haystack construction\.Each instance follows a "Needle\-in\-a\-Haystack" architecture\. TheHaystackconsists of 30\-day OHLCV histories ofBBsampled stocks formatted as price tables, interleaved with additional distractor stocks to increase retrieval difficulty\.
3. 3\.Needle construction\.TheNeedleconsists of specific investor transaction histories embedded within the context\. Each investor receivesDDcompleted buy–sell pairs drawn from distinct stocks, plus one open position \(the target stock\) shared across all investors\. The open position determines the dates on which the profit target can be achieved\.
4. 4\.Answer computation\.The reference answer and chain\-of\-thought are computed deterministically from the sampled prices and transactions\.
5. 5\.Quality filtering\.To limit null answers, the open transaction date is sampled from the first or last 25% of the time window\. Samples with no valid qualifying dates are retried with a new seed\.

## Appendix DConstruction of Expert Designed MAS

To establish a competitive upper bound for agentic performance on SMFR, we architect a manual MAS that utilizes structured decomposition and deterministic orchestration\. Unlike the automated frameworks discussed in Section[3](https://arxiv.org/html/2606.13003#S3), which rely on the LLM to discover and manage its own workflow, ourExpert\-MASenforces a strict separation between linguistic processing and logical control\.

#### Architecture and Role Specialization

Figure[4](https://arxiv.org/html/2606.13003#S3.F4)details the multi\-step pipeline designed to minimize context bloat and maximize sub\-task focus, composed of the following sub\-agents:

1. 1\.The Meta\-Agent:A specialized agent that acts as a structural parser, responsible for extracting the problem’s topology \(investor names, profit targets, and aggregation criteria\)\. This agent produces a structured JSON schema that drives the downstream orchestration, but performs no numerical reasoning itself\.
2. 2\.The ExtractorAgent:A reusable retrieval unit tasked with targeted information extraction from the 50k\+ token haystack\. It is prompted to locate specific transaction dates and prices as needed, effectively acting as a high\-precision filter\.
3. 3\.The CalculatorAgent:A numerical reasoning unit that computes realized P&L and derives target price thresholds\. By providing this agent only with the relevant extracted snippets, we ensure its reasoning window remains uncluttered by distractor tickers\.

#### Deterministic Orchestration and Parallelism

A significant departure from automated MAS is our use of aPython\-based Executorfor orchestration\. Rather than allowing the LLM to manage the "handoff" between agents, we utilize a deterministic control script\.

As shown in Figure[4](https://arxiv.org/html/2606.13003#S3.F4), the orchestrator dispatches sub\-tasks in parallel across the investor dimension\. While sequential dependencies are maintained within an investor’s logic chain \(e\.g\., Transactions→\\rightarrowP&L→\\rightarrowTarget Price\), the system executes the chains for allNNinvestors concurrently\. The final comparison and win\-determination are performed via deterministic Python logic\.

## Appendix EArchitectural Analysis

### E\.1DyLAN\[[22](https://arxiv.org/html/2606.13003#bib.bib105)\]

![Refer to caption](https://arxiv.org/html/2606.13003v1/plots/plot_roles.png)Figure 9:Highest ranked agents by importance score for different role settings in DyLAN\. Results are based on using GPT\-4o and GPT\-5 as backbone models for the GPQA\-Diamond task\. Plots reveal a slight positional bias towards the first agent regardless of role or backbone model\.To investigate the causal influence of role specialization in the remaining interactive cases, we compared three configurations: \(i\) task\-specific experts, \(ii\) random default roles, and \(iii\) generic assistant roles\. Surprisingly, the all\-assistant setting achieved the highest accuracy \(54\.41%\), outperforming task\-specific experts \(53\.40%\)\. Furthermore, rankings based on agent importance scores \(Figure[9](https://arxiv.org/html/2606.13003#A5.F9)\) reveal a persistent positional bias toward the first agent, regardless of assigned role or backbone model\. This suggests that the reported ‘MAS advantage’ in these paradigms is not a product of expert collaboration, but a byproduct of increased aggregate compute via redundant sampling\.

### E\.2AFlow\[[42](https://arxiv.org/html/2606.13003#bib.bib589)\]

The final MAS generated by AFlow on GPQA Diamond andSMFR\# ==================== GPQA Diamond Workflow \(GPT\-5\) ====================class Workflow:def \_\_init\_\_\(self,name: str,llm\_config,dataset: DatasetType,\) \-\> None:self\.name = nameself\.dataset = datasetself\.llm = create\_llm\_instance\(llm\_config\)self\.custom = operator\.Custom\(self\.llm\)self\.sc = operator\.ScEnsemble\(self\.llm\)async def \_\_call\_\_\(self, problem: str\):"""Implementation of the workflow"""solutions = \[\(await self\.custom\(input=problem, instruction=prompt\_custom\.QA\_BOX\_PROMPT\)\)\[’response’\] for i in range\(3\)\]vote = await self\.sc\(solutions=solutions\)return vote\[’response’\], self\.llm\.get\_usage\_summary\(\)\["total\_cost"\]\# ====================SMFRWorkflow \(GPT\-4o\) ====================class Workflow:def \_\_init\_\_\(self,name: str,llm\_config,dataset: DatasetType,\) \-\> None:self\.name = nameself\.dataset = datasetself\.llm = create\_llm\_instance\(llm\_config\)self\.custom = operator\.Custom\(self\.llm\)self\.sc\_ensemble = operator\.ScEnsemble\(self\.llm\)async def \_\_call\_\_\(self, problem: str\):"""Implementation of the workflow"""\# Generate multiple solutions using the custom operatorsolutions = \[\]for \_ in range\(3\): \# Generate three solutions for diversityresponse = await self\.custom\(input=problem, instruction=prompt\_custom\.XXX\_PROMPT\)solutions\.append\(response\[’response’\]\)\# Use ScEnsemble to select the most consistent solutionfinal\_solution = await self\.sc\_ensemble\(solutions=solutions\)return final\_solution\[’response’\], self\.llm\.get\_usage\_summary\(\)\["total\_cost"\]Figure 10:The final MAS workflows generated by AFlow on GPQA Diamond andSMFR, degenerating to trivial ensembling rather than sophisticated multi\-agent orchestration\.#### Degeneration to Trivial Ensembles\.

AFlow\[[42](https://arxiv.org/html/2606.13003#bib.bib589)\]is designed to discover sophisticated workflows via tree search over graph\-based code representations\. However, our inspection reveals a stark divergence from this objective: instead of manifesting complex coordination, the discovered MAS consistently degenerate into trivial ensembles\. As illustrated in our case analysis \(Figure[10](https://arxiv.org/html/2606.13003#A5.F10)\), “optimized” workflows frequently converge on a structure that simply iterates a single custom prompt three times before aggregation \- a configuration functionally identical to standard CoT\-SC\. Across 14 final workflows generated by GPT\-4o, GPT\-5, and GPT\-OSS\-120B on five datasets,50%50\\%\(7/147/14\) adopted this simplistic structure, with four of these actually underperforming the CoT\-SC baseline\.

### E\.3MAS\-Zero\[[19](https://arxiv.org/html/2606.13003#bib.bib207)\]

#### Verifier Bias and Consensus Collapse \(MAS\-Zero\)\.

In MAS\-Zero\[[19](https://arxiv.org/html/2606.13003#bib.bib207)\], a dedicated verifier agent aggregates outputs from parallel workers to select the optimal result\. We evaluate this mechanism across BrowseComp\-Plus, GPQA\-Diamond, HLE\-Math, andSMFRusing GPT\-4o and GPT\-5, with selection frequencies detailed in Fig[6](https://arxiv.org/html/2606.13003#S4.F6)\. Our analysis reveals a systematicpositional bias: the verifier disproportionately favors earlier entries in the context window, leading to prematureconsensus collapse\.

Across all benchmarks, we observe three consistent failure patterns:

1. 1\.Extreme Primacy:GPT\-4o exhibits a severe bias toward the initial block \(index 0, vanilla CoT\), selecting it in over 45% of instances, while CoT\-SC \(index 1\) remains a distant secondary choice\.
2. 2\.Broadened Initial Bias:GPT\-5 demonstrates a slightly more distributed but still front\-loaded preference, favoring the first four fundamental reasoning blocks \(indices 0–3\) while largely ignoring subsequent iterations\.
3. 3\.Blocks corresponding to later search rounds \(indices 4–8\) are rarely selected by either model, accounting for less than 15% of total selections combined\.

Consequently, the complex MAS architecture suffers from structural redundancy: subsequent worker agents function as "expensive witnesses", incurring full inference costs while exerting zero causal influence on the final output\.

### E\.4ADAS\[[13](https://arxiv.org/html/2606.13003#bib.bib174)\]

ADAS\[[13](https://arxiv.org/html/2606.13003#bib.bib174)\]optimizes MAS architectures through consecutive iterations of agent discovery\. While the framework is designed to iteratively refine performance, our analysis on GPQA\-Diamond reveals that the search results are non\-monotonic, lacking a consistent trajectory of improvement\. As illustrated in Figure[7](https://arxiv.org/html/2606.13003#S4.F7), validation accuracy frequently peaks early in the search phase before regressing or plateauing, rather than accumulating incremental gains\.

This differs from the pattern reported on the ARC dataset in the original work, where stronger\-performing agents were gradually discovered in later iterations\. We hypothesize this discrepancy stems from a potential evaluation artifact: through correspondence with the authors, we confirmed that their reported results were derived from evaluating all generated MAS candidates directly on the test set and selecting the global maximum\. Consequently, our findings suggest that ADAS functions primarily as a heuristic explorer of architectural variants rather than a reliable optimizer, where performance gains are susceptible to “lucky” iterations rather than structural evolution\.

#### Non\-monotonic Search Across Iterations\.

ADAS\[[13](https://arxiv.org/html/2606.13003#bib.bib174)\]aims to iteratively refine MAS architectures through automated agent discovery\. However, our analysis on GPQA\-Diamond reveals that architectural search is non\-monotonic: validation accuracy frequently peaks early and subsequently regresses or plateaus, rather than accumulating incremental gains \(see Figure[7](https://arxiv.org/html/2606.13003#S4.F7)\)\. This deviates from the original work’s reported performance on the ARC dataset, which we hypothesize is an artifact of selecting the global maximum from the test set across all candidates\.444Through correspondence with the authors, we confirmed their results were derived by evaluating all candidates directly on the test set\.These findings suggest that ADAS functions as a heuristic explorer rather than a reliable optimizer; performance gains appear to be the result of stochastic “lucky” iterations rather than a principled structural evolution toward superior reasoning\.

#### Architectural Redundancy\.

To isolate the structural drivers of performance, we conducted a motif analysis by mapping generated architectures to a rule\-based dictionary \(e\.g\., Self\-consistency, Aggregation, Verifier\)\. Across all settings, the primary positive signal originated from Self\-consistency motifs\. On GPQA\-Diamond, architectures incorporating these motifs achieved a mean accuracy of82\.19%82\.19\\%\(\+1\.34%\+1\.34\\%over the global average\), whereas “specialized” coordination motifs yielded negligible gains\. This mechanistic evidence confirms that automated search often converges on rediscovering CoT\-SC style sampling under more complex labels, rather than inventing novel or synergistic multi\-agent strategies\.

### E\.5MaAS\[[40](https://arxiv.org/html/2606.13003#bib.bib598)\]

#### Incentive Misalignment and Routing Collapse\.

MaAS\[[40](https://arxiv.org/html/2606.13003#bib.bib598)\]optimizes its controller via Monte Carlo gradient estimation, balancing an accuracy objective against a cost penalty\. However, we find that with highly capable base models \(e\.g\., GPT\-5\), accuracy frequently saturates, flattening the gradient to∼1/K\\sim 1/Kand extinguishing the signal required to learn task\-specific routing\. Consequently, the controller’s behavior is dictated almost entirely by the cost term, resulting in two distinct failure modes: \(1\)Cost\-Minimizing Collapseon BrowseComp\-Plus, where high cost variance drives the controller toward a trivial single I/O call \(74\.2%74\.2\\%of activations\); and \(2\)Stochastic Stallingon GPQA\-Diamond, where negligible cost differentials trap the controller in its initialized near\-uniform distribution\. In both cases, the supernet fails to acquire meaningful routing logic, settling into either a “cheap shortcut” or an undifferentiated ensemble that consistently underperforms independent CoT\-SC sampling \(Table[2](https://arxiv.org/html/2606.13003#S4.T2)\)\.

### E\.6MAS\-Orchestra\[[18](https://arxiv.org/html/2606.13003#bib.bib209)\]

#### Policy Collapse in Dynamic Orchestration\.

MAS\-Orchestra\[[18](https://arxiv.org/html/2606.13003#bib.bib209)\]is designed to perform dynamic resource allocation by routing queries to agents based on task difficulty\. However, our analysis reveals a total policy collapse into difficulty\-agnostic behavior\. Across all benchmarks, the system largely ignores its diverse agent pool, converging instead on a rigid binary preference for high\-overhead Debate and Reflexion agents \(see Table[2](https://arxiv.org/html/2606.13003#S4.T2)\)\. Crucially, the orchestrator fails to scale agent complexity to task difficulty; despite GPQA\-Diamond posing a lower reasoning ceiling than HLE\-Math, the system exhibited a higher reliance on Debate agents for the former \(84\.9%84\.9\\%\) than the latter \(79\.2%79\.2\\%\)\. These results demonstrate that the orchestrator does not manifest adaptive configuration; instead of learning task\-specific strategies, it settles into a static, greedy preference for maximum\-overhead sub\-agents regardless of a query’s actual requirements\.

## Appendix FScope and Limitations

#### Model Diversity and Selection Bias\.

Our study primarily utilizes frontier models from the OpenAI and Google families, alongside a single representative open\-source backbone\. While this selection spans varying scales and generations, it is possible that specific architectural idiosyncrasies of other model families \(e\.g\., Anthropic’s Claude or Meta’s Llama series\) might yield different interaction dynamics\. Furthermore, because cost\-efficiency was a central pillar of our evaluation, we did not explore infinite\-budget regimes where extremely large ensembles might eventually overcome the identified positional biases through sheer scale\.

#### Reasoning vs\. Tool\-Use Proficiency\.

Our evaluation focuses primarily on cognitive orchestration and long\-horizon reasoning within closed or semi\-closed contexts\. While benchmarks like BrowseComp\-Plus and SWE\-bench Lite involve retrieval and patch generation, we did not evaluate the broader spectrum of autonomous tool\-use, such as real\-time API interaction, multi\-modal sensor integration, or complex shell environments\. It remains possible that the structural efficiencies identified in our "Expert\-MAS" might differ in environments where the primary bottleneck is external tool\-call latency or protocol adherence rather than internal logical consistency\. Our findings of functional collapse are therefore most applicable to reasoning\-heavy agentic workflows\.

#### Optimization Hyperparameters\.

Our evaluation of automated frameworks \(e\.g\., ADAS, AFlow\) utilized the default search hyperparameters provided by the original authors\. It is conceivable that with extensive, domain\-specific hyperparameter tuning, these frameworks could find more robust coordination motifs\. However, we intentionally maintain default configurations across all systems \- including CoT\-SC and Expert\-MAS \- to evaluate out\-of\-the\-box reliability\. Our findings suggest that while expert\-designed and simple SAS baselines remain robust under default settings, current automated MAS search processes are highly sensitive, failing to consistently outperform SAS without extensive optimization\.

## Appendix GBroader Impacts

This work introduces a diagnostic benchmark designed to evaluate the reasoning efficiency of multi\-agent systems\. While the dataset utilizes financial market primitives, it is intended strictly for AI safety and architectural research and is not validated for real\-world financial forecasting or automated trading\. By identifying structural bloat in AI workflows, this research promotes the development of more computationally efficient and transparent models, potentially reducing the environmental and economic costs of large\-scale AI deployment\. We do not foresee any significant negative societal impacts, provided the benchmark is used as a diagnostic tool rather than a predictive model for safety\-critical domains\.