SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
Summary
SMAC-Talk is a new benchmark that extends the StarCraft Multi-Agent Challenge to evaluate LLM-based agents in cooperative multi-agent environments with natural language communication. It includes scenarios with deceptive communicators and benchmarks agents using models from the Qwen3.5 family to study how reasoning, memory, and scale affect coordination.
View Cached Full Text
Cached at: 06/05/26, 02:05 AM
# SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models Source: [https://arxiv.org/abs/2606.04202](https://arxiv.org/abs/2606.04202) [View PDF](https://arxiv.org/pdf/2606.04202)[HTML \(experimental\)](https://arxiv.org/html/2606.04202v1) > Abstract:As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation\. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty\. We introduce SMAC\-Talk, a natural language extension of the StarCraft Multi\-Agent Challenge for evaluating LLM\-based agents in cooperative multi\-agent environments\. The environment has several key features such as decentralized control, partial observability and long\-horizon decision making\. SMAC\-Talk includes a natural language communication channel which is used to probe agent coordination and trust\. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone\. We provide three agents for benchmarking using 4 models from the Qwen3\.5 family and study how reasoning structure, memory and model scale affect coordination between agents\. We release SMAC\-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi\-agent settings\. ## Submission history From: Joel Sol \[[view email](https://arxiv.org/show-email/d54f9fd3/2606.04202)\] **\[v1\]**Tue, 2 Jun 2026 20:40:04 UTC \(866 KB\)
Similar Articles
Exploring Collaboration between a language and a non-language agent
The paper introduces LLAMIA-Bench, a benchmark for collaborative chess tasks between language models and non-language agents, and proposes latent state internalization to outperform text-based verbalization, with a 14B model matching or exceeding frontier models like GPT-5.1.
New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]
Introduces a new benchmark for evaluating multi-agent coordination in LLMs, finding that most models struggle with long-horizon open-ended tasks, but Gemini 3.1 Pro performs comparably to trained MARL agents on the hardest setting.
TMAS: Scaling Test-Time Compute via Multi-Agent Synergy
TMAS introduces a multi-agent framework that enhances large language model reasoning by scaling test-time compute through structured collaboration and hierarchical memory systems. The approach uses specialized agents, cross-trajectory information flow, and hybrid reward reinforcement learning to improve iterative scaling and stability on challenging reasoning benchmarks.
A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
This paper introduces a tri-agent framework for evaluating and aligning the question clarification capabilities of large language models, using three LLM-based agents to simulate and assess clarification dialogues.
Decentralized Multi-Agent Systems with Shared Context
This paper introduces Decentralized Language Models (DeLM), a framework for multi-agent systems that uses parallel agents with a shared verified context to improve test-time scaling and reduce costs, achieving state-of-the-art results on SWE-bench Verified and LongBench-v2.