Exploring Collaboration between a language and a non-language agent
Summary
The paper introduces LLAMIA-Bench, a benchmark for collaborative chess tasks between language models and non-language agents, and proposes latent state internalization to outperform text-based verbalization, with a 14B model matching or exceeding frontier models like GPT-5.1.
View Cached Full Text
Cached at: 09/03/26, 11:51 AM
Paper page - Exploring Collaboration between a language and a non-language agent
Source: https://huggingface.co/papers/2609.00474
Abstract
A benchmark of collaborative chess tasks shows that integrating continuous subagent representations directly into language models via learned state tokens outperforms text-based verbalization and scales effectively.
LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integratingnon-language agentswith LLMs would requireverbalization: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whetherverbalizationconstitutes a bottleneck, we introduceLLAMIA-Bench, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration withnon-language agents, we introducelatent state internalization, which projects the subagent’s continuous representations directly into the LLM’s token stream aslearned state tokens, withdynamic re-encodingas actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistentverbalization debt: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, LLAMIA, trained withlatent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2609\.00474
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.00474 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.00474 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.00474 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Exploring Collaboration between a language and a non-language agent
This paper introduces LLAMIA-Bench and a method called latent state internalization to enhance collaboration between language models and non-language agents, showing that internalizing continuous representations outperforms verbalization and matches frontier models like GPT-5.1.
SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
SMAC-Talk is a new benchmark that extends the StarCraft Multi-Agent Challenge to evaluate LLM-based agents in cooperative multi-agent environments with natural language communication. It includes scenarios with deceptive communicators and benchmarks agents using models from the Qwen3.5 family to study how reasoning, memory, and scale affect coordination.
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
CollabBench is a new benchmark for evaluating and training LLM agents in cooperative games, featuring diverse player simulation and a collaborative training paradigm. Experiments show 19.5% higher efficiency and 24.4% improved affective performance over base models.
New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]
Introduces a new benchmark for evaluating multi-agent coordination in LLMs, finding that most models struggle with long-horizon open-ended tasks, but Gemini 3.1 Pro performs comparably to trained MARL agents on the hardest setting.
Large Language Models over Networks: Collaborative Intelligence under Resource Constraints
This paper explores collaborative intelligence paradigms where distributed Large Language Models work together across devices and clouds to handle resource constraints. It covers vertical device-cloud collaboration, horizontal multi-agent collaboration, routing policies, and open research challenges in scalable and trustworthy cooperative AI.