New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]

Reddit r/MachineLearning Papers
multi-agent coordination benchmark llm language-agents open-ended evaluation

Summary

Introduces a new benchmark for evaluating multi-agent coordination in LLMs, finding that most models struggle with long-horizon open-ended tasks, but Gemini 3.1 Pro performs comparably to trained MARL agents on the hardest setting.

Can LLM agents coordinate in long-horizon, open-ended worlds? We evaluate 13 modern LLMs in a new benchmark where agents must work together to explore, communicate, trade resources, craft tools, build structures, and fight mobs. TL;DR: Most agents struggle, averaging only ~6% normalised return. Yet on the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps. More broadly, we find coordination is a distinct bottleneck beyond long-horizon task competence, with communication having the largest effect in our harness ablations. Paper: https://arxiv.org/abs/2606.08340 Project page and leaderboard: https://alem-world.github.io Code: https://github.com/alem-world/alem-env Interactive traces: https://alem-world.github.io/traces.html Feel free to ask any questions!
Original Article

Similar Articles

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

Hugging Face Daily Papers

AgentWorld introduces a benchmark for evaluating long-horizon collaboration in multi-agent LLM systems with 100 tasks and a new causal collaboration effectiveness metric. Experiments show that even top models achieve only 52% task success, highlighting failure modes like communication breakdowns.

Multi-Agent LLMs Fail to Explore Each Other

Hugging Face Daily Papers

This paper identifies that current LLM agents fail to systematically explore their peers, leading to poor coordination, and introduces MACE, a lightweight framework using contextual bandits for effective peer selection.