Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration

# Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration

Source: https://arxiv.org/html/2507.02935

###### Abstract

Successful human-agent teaming relies on an *agent* being able to understand instructions given by a (human) *principal*. In many cases, an instruction may be incomplete or ambiguous. In such cases, the agent must infer the unspoken intentions from their shared context, that is, it must exercise the principal's Theory of Mind (ToM) and infer the mental states of its principal. We consider the prospects of effective human-agent collaboration using large language models (LLMs). To assess ToM in a dynamic, goal-oriented, and collaborative environment, we introduce a novel task, *Instruction Inference*, in which an *agent* assists a *principal* in reaching a goal by interpreting incomplete or ambiguous instructions. We present *Tomcat*, an LLM-based agent, designed to exhibit ToM reasoning in interpreting and responding to the principal's instructions. We implemented two variants of Tomcat. One, dubbed Fs-CoT (Fs for few-shot, CoT for chain-of-thought), is based on a small number of examples demonstrating the requisite structured reasoning. One, dubbed CP (commonsense prompt), relies on commonsense knowledge and information about the problem. We realized both variants of Tomcat on three leading LLMs, namely, GPT-4o, DeepSeek-R1, and Gemma-3-27B. To evaluate the effectiveness of Tomcat, we conducted a study with 52 human participants in which we provided participants with the same information as the CP variant. We computed intent accuracy, action optimality, and planning optimality to measure the ToM capabilities of Tomcat and our study participants. We found that Tomcat with Fs-CoT, particularly with GPT-4o and DeepSeek-R1, achieves performance comparable to the human participants, underscoring its ToM potential for human-agent collaboration.

###### Keywords

Theory of Mind; Human-Agent Collaboration; Large Language Models; Instruction Interpretation; Ambiguity

††journal: Artificial Intelligence
\\affiliation [ncsu] organization=North Carolina State University, city=Raleigh, state=NC, country=USA
\\affiliation [tud] organization=Delft University of Technology, city=Delft, country=The Netherlands

## 1 Introduction

Natural communication is rarely limited to literal language[1,2]. Instead, collaborators frequently use indirect expressions whose intended meaning must be inferred from contextual cues and shared task understanding[3,4,5]. Successful collaboration therefore presupposes an ability to interpret incomplete and ambiguous language, which in turn presupposes an understanding of the collaborator's mental state[6,7]. The capability of modeling another entity as endowed with a *mind*, e.g., beliefs and intentions, is referred to as having a Theory of Mind (ToM)[8,9]. ToM enables people to infer unspoken intentions and respond appropriately to indirect cues[10].

\\includestandalone [width=mode=image]tikz/motivation-unclear-1
\\includestandalone [mode=image,width=]tikz/motivation-unclear-2

Figure 1: Interpreting unclear instructions by traditional vs. LLM agents where the latter performs the task considering the pragmatic context.

For instance, a collaborator may say "Can you get that?" while pointing to an object (e.g., a key) in the shared environment. Interpreting such an instruction requires inferring which object was requested based on contextual cues and shared task understanding, thereby requiring ToM reasoning. At the first-order ToM level, this may involve reasoning such as "I believe that you want the red key." In more complex situations, collaborators may engage in higher-order reasoning, i.e., second-order ToM, where one reasons about another party's beliefs, for instance, "I believe that you believe that I know which key is needed." Such higher-order reasoning is often observed in strategic interactions involving persuasion, negotiation, or deception[11,12,13]. However, in cooperative settings, resolving unclear instructions often relies on reasoning about the collaborator's immediate goals and intentions, which can be captured through first-order ToM[4,6,14].

Equipping AI agents with ToM can foster more natural and effective human-agent interaction[15], particularly in settings where communication is incomplete or ambiguous[4]. However, traditional autonomous agents have historically struggled to interpret such language. Prior work in instruction following and human-agent collaboration shows that agents lacking pragmatic context[16,17,18] tend to interpret instructions literally and thus fail when instructions are incomplete or ambiguous[3,5]. As a result, these agents often cannot infer a collaborator's intended goal from contextual cues in dynamic task environments, as shown in Figure 1.

The advent of large language models (LLMs) offers a promising avenue to bridge this gap. LLMs have demonstrated an emergent capability to account for pragmatic context and perform sophisticated linguistic reasoning[19,20,21,22]. Recent studies suggest that these models may exhibit ToM capabilities by modeling the mental states of others[23,20]. However, evaluations of the ToM capabilities of LLMs rely on static benchmarks such as false-belief, faux-pas, or conversational reasoning tasks. For example, Strachan et al.[19] measure the ToM capabilities of LLMs by providing a short conversational vignette followed by targeted questions to assess how LLMs reason about another party's mental states. These settings evaluate reasoning about mental states but do not test whether agents can use such reasoning to interpret *unclear instructions* and act appropriately in *collaborative task environments*, as shown by the contrast in Figure 1.

To address this challenge, we introduce a new ToM task, *Instruction Inference*, consisting of 20 distinct problem scenarios. Each scenario evaluates an agent's capability to infer a principal's (simulated human) intended goal from clear and unclear instructions and act appropriately in a collaborative environment. We use the term *instructions* to refer to commands, requests, or directives issued by the principal. To ground the design of our dataset, we draw on Grice's theory of conversational implicature[1]. Grice identifies that cooperative communication is guided by four conversational maxims: Quantity, Quality, Relation, and Manner. Violations of these maxims trigger pragmatic inference, requiring listeners to infer the speaker's intended meaning beyond the literal utterance. We distinguish two types of unclear instructions grounded in this framework. *Incomplete* instructions omit critical information required to determine the intended action; they correspond to violations of Quantity maxim. *Ambiguous* instructions yield two or more possible interpretations; they correspond to violations of Manner maxim. We focus on these two categories because violations of Quantity and Manner maxims commonly arise in cooperative settings where collaborators assume shared context and task understanding[3,4,5]. By contrast, violations of the Quality and Relation maxims are less characteristic of cooperative collaboration, as they typically correspond to deception or conversational breakdown.

We implement this task using *Tomcat*, an LLM-based agent deployed in a collaborative environment called Doors, Keys, and Gems[3], shown in Figure 3. In this environment, Tomcat assists its principal in achieving their objective of collecting a gem by retrieving keys or unlocking doors. Tomcat interprets both clear and unclear instructions by reasoning about the task environment and the principal's movements. Our study employs two variants of Tomcat, *Few-shot Chain-of-thought (Fs-CoT) Prompt* and *Commonsense Prompt (CP)* (Figure 2). Tomcat with Fs-CoT leverages in-context learning[24] and few-shot chain-of-thought (Fs-CoT) prompting[25] to incorporate task knowledge and contextual cues to infer the principal's intended goal. Tomcat with CP relies on commonsense knowledge about the Instruction Inference task with two detailed problem walkthroughs.

Refer to caption

Figure 2: Pipeline of the Instruction Inference task with Tomcat and participants.

We realized both variants of Tomcat on three LLMs—GPT-4o, DeepSeek-R1, and Gemma-3-27B—and compare their ToM capabilities with human participants. To establish a benchmark for comparison, we conduct an empirical study with 52 human participants who perform the same Instruction Inference task under identical conditions as Tomcat with CP, i.e., using the same information and objectives. This evaluation is framed by the following research questions.

**RQ1:** How do Tomcat's ToM capabilities on Instruction Inference compare to human performance?

**RQ2:** Does Tomcat with Fs-CoT outperform Tomcat with CP on Instruction Inference?

To summarize, our **contributions** are as follows:

1. **Instruction Inference task**: We introduce Instruction Inference, a novel task for evaluating ToM in a collaborative environment where instructions may be clear or unclear (incomplete or ambiguous). Unlike existing ToM benchmarks that rely primarily on narrative or conversational scenarios, Instruction Inference evaluates whether an agent can infer a principal's intended goal from clear and unclear instructions and act appropriately in a shared task environment.

2. **Tomcat agent for instruction inference**: We develop *Tomcat*, an LLM-based agent that performs instruction inference in a dynamic environment by reasoning about pragmatic context and the principal's actions. We investigate two variants of Tomcat: a *Few-shot Chain-of-Thought (Fs-CoT)* variant using seven reasoning exemplars, and a *Commonsense Prompt (CP)* variant that relies on task knowledge with two problem walkthroughs.

3. **Empirical evaluation with human participants**: We conduct an empirical evaluation comparing both Tomcat variants' ToM capabilities, realized on three LLMs (GPT-4o, DeepSeek-R1, and Gemma-3-27B), with those of 52 human participants on our Instruction Inference task. Our results show that the Fs-CoT variants of GPT-4o and DeepSeek-R1 achieve performance comparable to human participants.

The subsequent sections provide a systematic analysis of each module in our pipeline, as shown in Figure 2.

## 2 Related Work

#### ToM Benchmarks

Building on the recent success in LLMs, Strachan et al.[19] evaluate models like GPT-4, GPT-3.5, and Llama2-70B on ToM tasks including false-beliefs, faux pas, irony, strange stories, and hinting tasks. The tasks adopt practical conversational social scenarios in contrast to our goal-driven collaborative setting. Strachan et al.[19] report that GPT-4 outperforms humans in irony, hinting, and strange stories, but not in faux pas. Similarly, Kosinski[20] tests foundational ToM concepts, such as belief attribution and reasoning about mental states, using structured cognitive frameworks, revealing that advanced LLMs like GPT-4 exhibit human-like reasoning in certain tasks but not in nuanced scenarios. Kim et al.[26] introduce *FANToM*, a conversational benchmark that evaluates mental state reasoning under asymmetric information. Chan et al.[27] present *NegotiationToM*, a benchmark grounded in the belief-desire-intention (BDI) framework that evaluates ToM reasoning in negotiation dialogues derived from the CaSiNo dataset[28]. Gandhi et al.[21] introduce BigToM, a large scale benchmark that extends classical narrative-based ToM tasks to evaluate belief reasoning across diverse social scenarios.

#### Higher-order ToM Reasoning

Street[29] posits that LLMs' ToM capabilities can help improve human-agent collaboration. Street et al.[30] develop the Multi-Order Theory of Mind Q&A (MoToMQA) test suite to evaluate LLMs on ToM tasks that require recursive reasoning about nested mental states (second- to sixth-order). They show GPT-4's capability to handle the deep recursive reasoning that is critical for understanding complex social interactions. Jamali et al.[31] analyze hidden embeddings in LLMs and identify neuron-like units responsive to true- and false-belief tasks, drawing parallels to ToM processing by humans.

#### ToM in Multiagent Systems

Recent work has expanded ToM into more complex domains, such as multiagent systems, automated planning, and multimodal environments. Rocha et al.[32] review the use of ToM in multiagent systems, noting its value for improving coordination, generating explanations, providing assistance, and modeling deception. In planning, Shvo et al.[33] use ToM to help agents identify conflicting beliefs about a plan's validity and correct these misunderstandings through direct communication or physical actions. Shi et al.[34] introduce MuMA-ToM, a multimodal multiagent benchmark for inferring an agent's beliefs and social goals in everyday household activities using video and textual context. Yim et al.[35] adopt the cooperative card game *Guandan* and show that ToM-based planning helps LLM agents better coordinate with allies and adapt strategies against opponents.

Table 1: Comparison of ToM benchmarks across format, modality, collaboration setting, and instruction type.

#### ToM in Multiagent Reinforcement Learning