New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]

Reddit r/MachineLearning Papers
multi-agent coordination benchmark llm language-agents open-ended evaluation

Summary

Introduces a new benchmark for evaluating multi-agent coordination in LLMs, finding that most models struggle with long-horizon open-ended tasks, but Gemini 3.1 Pro performs comparably to trained MARL agents on the hardest setting.

Can LLM agents coordinate in long-horizon, open-ended worlds? We evaluate 13 modern LLMs in a new benchmark where agents must work together to explore, communicate, trade resources, craft tools, build structures, and fight mobs. TL;DR: Most agents struggle, averaging only ~6% normalised return. Yet on the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps. More broadly, we find coordination is a distinct bottleneck beyond long-horizon task competence, with communication having the largest effect in our harness ablations. Paper: https://arxiv.org/abs/2606.08340 Project page and leaderboard: https://alem-world.github.io Code: https://github.com/alem-world/alem-env Interactive traces: https://alem-world.github.io/traces.html Feel free to ask any questions!
Original Article

Similar Articles

Multi-Agent LLMs Fail to Explore Each Other

Hugging Face Daily Papers

This paper identifies that current LLM agents fail to systematically explore their peers, leading to poor coordination, and introduces MACE, a lightweight framework using contextual bandits for effective peer selection.

COOPA: A Modular LLM Agent Architecture for Operations Research Problems

arXiv cs.LG

This paper introduces COOPA, a modular LLM agent architecture for operations research problems that combines iterative confidence-based modeling, element-level provenance, and multi-solver routing. Evaluated across eight LLM backbones and four baselines, COOPA achieves the best macro-average accuracy on six backbones and improves over the strongest baseline by up to 6.7 percentage points.

GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models

arXiv cs.AI

The paper introduces GeoNatureAgent Benchmark, the first benchmark for evaluating LLM agents on environmental geospatial analysis tasks via structured tool calls. It evaluates seven models on 93 tasks across 18 categories and finds Claude Sonnet 4 achieves highest accuracy at 60.8%, while open-weight models like DeepSeek V3.2 offer strong cost-performance tradeoffs.