Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning

# Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
Source: [https://arxiv.org/html/2608.04646](https://arxiv.org/html/2608.04646)
11institutetext:LIACS, Leiden University, The Netherlands11email:i\.b\.de\.haan\.2@umail\.leidenuniv\.nl,11email:p\.w\.h\.van\.der\.putten@liacs\.leidenuniv\.nl, m\.j\.van\.duijn@liacs\.leidenuniv\.nl###### Abstract

Large language models \(LLMs\) have recently shown strong performance on Theory of Mind \(ToM\) tests, prompting debate about the nature and validity of the underlying capabilities\. At the same time, reasoning\-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks\. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks\. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations\. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation\. We read this as evidence for a robustness\-based account rather than for a new ToM\-specific ability\.

## 1Introduction

In recent years, large language models \(LLMs\) have become sufficiently capable to be part of our daily lives, and are increasingly used in social and agentic settings\[[13](https://arxiv.org/html/2608.04646#bib.bib67)\]\. Users tend to attribute human\-like intentionality and reasoning to those models, even though the extent to which the models are capable of complex reasoning remains, to a large extent, an open research problem\[[29](https://arxiv.org/html/2608.04646#bib.bib10),[16](https://arxiv.org/html/2608.04646#bib.bib9),[23](https://arxiv.org/html/2608.04646#bib.bib69),[14](https://arxiv.org/html/2608.04646#bib.bib68)\]\.

Due to the complexity of these systems, even if all information about their architecture is known, it is still difficult to determine and predict their actual behavior\. Considering this, researchers in a multitude of fields started investigating those systems and their intelligence by interacting with them and understanding their behavior, rather than by analyzing their architecture\.\[[16](https://arxiv.org/html/2608.04646#bib.bib9)\]

In this context, previous work has debated the extent to which LLMs demonstrate Theory of Mind \(ToM\) behavior, that is, the ability to reason about mental states, beliefs, intentions, and desires\. Some early claims were optimistic of such a skill emerging to some extent\[[9](https://arxiv.org/html/2608.04646#bib.bib61)\]\. However, follow\-up criticism\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\]led to the creation of benchmarks testing these abilities more comprehensively\[[22](https://arxiv.org/html/2608.04646#bib.bib2),[24](https://arxiv.org/html/2608.04646#bib.bib12),[8](https://arxiv.org/html/2608.04646#bib.bib30),[4](https://arxiv.org/html/2608.04646#bib.bib5),[6](https://arxiv.org/html/2608.04646#bib.bib1)\]\.

Since then, so\-called reasoning model families, like OpenAI’s GPT\-5, Anthropic’s Claude, and the DeepSeek R series, have been released\. The innovation behind these models is that they are trained, via reinforcement learning, to “think before answering”, that is, they are trained to produce a complex chain of thought before answering the query, a form of inference\-time scaling\[[11](https://arxiv.org/html/2608.04646#bib.bib14),[12](https://arxiv.org/html/2608.04646#bib.bib62),[5](https://arxiv.org/html/2608.04646#bib.bib65),[14](https://arxiv.org/html/2608.04646#bib.bib68)\]\. This new type of model might not only perform significantly better in ToM tasks, but the chain\-of\-thought could, in theory, yield important insights about what those models are actually doing to answer the queries\.

However, it remains unclear whether these improvements reflect fundamentally stronger ToM reasoning capabilities, or instead improved robustness in finding correct solutions under prompt and task variations\. Understanding this distinction is important for interpreting apparent progress in social\-cognitive abilities of LLMs and for designing reliable evaluation protocols\.

This paper studies the role of reasoning in the performance of large language models on ToM tasks, and can be seen as a contribution of humanities and social sciences to AI, and vice versa, though we are only exploring LLM behavior and explicitly not making any strong AI claims about whether LLMs truly possess ToM\. First, we evaluate recent reasoning\-oriented models using established psychological ToM tests from the battery of\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\]\. Second, we introduce prompt variants inspired by\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\]designed to probe robustness under task\-preserving prompt perturbations\. Third, we compare reasoning\-enabled and non\-reasoning model behavior across these tests to examine how reasoning affects performance stability\. Finally, we complement our experiments with benchmark results reported in\[[7](https://arxiv.org/html/2608.04646#bib.bib35)\]\. All prompts, data, and code are publicly available at[this GitHub repository](https://github.com/ianbdehaan/Evaluating_ToM_in_Reasoning_Models), including a supplementary materials document\.

The remainder of this paper is structured as follows: Section[2](https://arxiv.org/html/2608.04646#S2)provides an overview of related work on the subject\. Section[3](https://arxiv.org/html/2608.04646#S3)introduces the materials and methods employed in the experiments, and Section[4](https://arxiv.org/html/2608.04646#S4)presents the experimental results; Section[5](https://arxiv.org/html/2608.04646#S5)interprets the obtained results, points out some limitations of the current work, and suggests new directions in which the research can be expanded; Section[6](https://arxiv.org/html/2608.04646#S6)sums up the findings\.

## 2Background

In this section, a bibliographical review in the context of the research done is presented\. The contents covered are Section[2\.1](https://arxiv.org/html/2608.04646#S2.SS1), a review of what chains of thought are and the recent reinforcement learning with verifiable rewards \(RLVR\) trained reasoning models, and Section[2\.2](https://arxiv.org/html/2608.04646#S2.SS2), an introduction on the topic of ToM, how it would be a useful skill for LLMs to have, and in Section[2\.3](https://arxiv.org/html/2608.04646#S2.SS3)third\-party benchmarks investigating to which extent they do\.

### 2\.1Chain of Thought and Reasoning Models

RLVR proved itself to be effective at inducing complex reasoning in LLMs and, using this technique, a new wave of models, refined to produce chains of thoughts together with the answer, started emerging and were named*reasoning models*\[[27](https://arxiv.org/html/2608.04646#bib.bib43),[14](https://arxiv.org/html/2608.04646#bib.bib68),[13](https://arxiv.org/html/2608.04646#bib.bib67)\]\. The implementation details of those models are an important topic, but outside of the scope of this paper\.

However, although promising results in several fields, particularly in mathematics and programming, ignited hope that RLVR\-trained LLMs could achieve a more general form of intelligence,\[[28](https://arxiv.org/html/2608.04646#bib.bib23)\]demonstrated that, in reality, the gained performance of those models comes from a more efficient sampling of answers on problems that are already solvable by the base model, not introducing fundamentally new reasoning capabilities\. Therefore, the reasoning capacity of a reasoning model remains bounded by its base model\. Reasoning was also initially viewed as promising in the way towards tracking reward hacking in LLMs, but it has been shown that, at least in scenarios where the CoT is not necessary for such behavior, the verbalization of it is very low\[[3](https://arxiv.org/html/2608.04646#bib.bib31)\]\. This finding also makes one conclude, more generally, that reasoning models don’t always include their true reasoning in their think tokens, although the same article also showed that, at least part of the time, they do\.

### 2\.2Theory of Mind and LLMs

ToM is generally defined as the ability of an individual to attribute mental states to others \(including oneself\) and keep track of them, and also tightly relate it to the notion of self and others\[[2](https://arxiv.org/html/2608.04646#bib.bib55),[24](https://arxiv.org/html/2608.04646#bib.bib12)\], additional definitions in supplementary materials in the repository\. According to\[[1](https://arxiv.org/html/2608.04646#bib.bib59)\], “The ability to make inferences about what other people believe to be the case in a given situation allows one to predict what they will do”, this notion not only shows the importance that ToM has in human social interactions, but also allows one to design experiments to test whether an agent demonstrates ToM behavior or not\. Quesque and Rossetti\[[15](https://arxiv.org/html/2608.04646#bib.bib44)\]suggest that, for a particular task to act as a valid ToM assessment, it should follow two criteria: a task should not only involve attributing mental states to others, but those should be different from their own \(*‘non\-merging’ property*and an easier process than ToM should not be able to account for success in a particular task*“mentalizing” property*\.

Those two principles show themselves in many different tasks, such as understanding pretend play, bluffing, white lies, etc\., but one particular type of task became the litmus test of the field: the false belief test\. Such a test involves understanding that an agent, when operating under incorrect information, will act according to it and not to the real state of the world\[[17](https://arxiv.org/html/2608.04646#bib.bib24)\]\.

With the popularization of LLMs, it became key for those systems to have the necessary skills to engage in social interactions\. In this context, researchers started discussing the extent to which they possess ToM\.\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\]showed that, as of 2023, most LLMs operated below the performance of children aged 7\-10 in some LLM tasks\.\[[19](https://arxiv.org/html/2608.04646#bib.bib27)\]demonstrated that GPT\-4 and Flan\-PaLM achieve adult or near\-adult performance in higher\-order ToM tasks\.

However, there is ongoing debate about whether the success of LLMs in those ToM tasks actually signifies that they truly developed ToM skills\[[18](https://arxiv.org/html/2608.04646#bib.bib60)\]\.\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\]demonstrates that, although GPT\-3 passes ToM tasks, alterations in prompts that seem trivial to us cause it to fail on them\. The author claims that this means that GPT3’s understanding of the underlying principle behind ToM is not real\.

In the context of this paper, the concept of a real skill for AI regards simply what it shows via our interactions with it, so actingas ifit had a skill orreallyhaving the skill are treated as equivalent\. This is the case as claims about the true nature of AI’s phenomenology and the possibility of it being a philosophical zombie, although valid and interesting discussions, are well beyond the scope of this paper, so Turing’s more instrumentalist approach is adopted\[[20](https://arxiv.org/html/2608.04646#bib.bib66)\]\. It is important, however, to point out that it still makes sense to make claims like the one from\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\], as the lack of ‘reality’ in the AI understanding presents itself through properties of the model’s response\.

Despite the discussion surrounding the validity of using the frequency of success in ToM tasks as a real assessment of a model having ToM skills, several benchmarks started to appear, each with a specific approach in the composition of the tasks\.\[[8](https://arxiv.org/html/2608.04646#bib.bib30)\]With benchmarks in place, it became easy to test if any particular technique could improve models’ performance in ToM tasks\. The first thing tested was the impact of CoT prompting on the models\. It was found to increase their performance at least in a few cases\[[8](https://arxiv.org/html/2608.04646#bib.bib30)\]\. Also, other authors proposed other approaches, which include but are not limited to:

- •*Perspective taking*prompts models to first filter the context to only what characters know before answering the ToM task\[[25](https://arxiv.org/html/2608.04646#bib.bib47)\]\.
- •*DEL\-ToM*is an approach that constitutes scaling the inference time of models through an approach grounded in dynamic epistemic logic\[[26](https://arxiv.org/html/2608.04646#bib.bib6)\]\.
- •*Thought Tracing*is an inference\-time reasoning algorithm that keeps track and weights different hypotheses surrounding characters’ mental state in ToM tasks\[[7](https://arxiv.org/html/2608.04646#bib.bib35)\]\.

A useful concept to shine additional light on ToM tasks is the one of Orders of Reasoning\. To illustrate what this concept means, imagine three characters, Adu, Lin, and Lily\. If Adu thinks "Lin is cool", this is considered a 0th\-degree reasoning, as he is simply thinking about Lin, not addressing his mental state\. On the other hand, if he thinks "Lilly thinks that Lin is cool", that would be a 1st\-degree reasoning\. To additionally illustrate higher degree reasoning, consider that Adu thinks "4\) Lily thinks that 3\) I think that 2\) she thinks the 1\) Lin thinks that 0\) she’s cool"\. In short, the order of reasoning of a ToM task is recursively how many mental representations it requires\.\[[10](https://arxiv.org/html/2608.04646#bib.bib33)\]

### 2\.3Third\-Party Benchmark Results

Efforts have already been made to test reasoning and non\-reasoning models directly on established ToM benchmarks\. Most relevant to us,\[[7](https://arxiv.org/html/2608.04646#bib.bib35)\]report results for a range of models, several reasoning models alongside their non\-reasoning peers, across the FANToM, BigToM, MMToM\-QA, and ParaphrasedToMi benchmarks\. We reproduce their numbers in Table[1](https://arxiv.org/html/2608.04646#S2.T1)for reference\. Note that these models are slightly older than the ones used in our own experiments\.

Two patterns are worth highlighting, which we return to in Section[5](https://arxiv.org/html/2608.04646#S5): reasoning models tend to score above comparable non\-reasoning models on most benchmarks, and the gap is largest on FANToM, where a model only scores if it answers every question of a given type correctly, so inconsistency is penalised heavily\. Both observations are consistent with the robustness\-based reading we develop through the paper, and we treat them as supporting external evidence\.

Table 1:Third party ToM benchmark results for reference, from\[[7](https://arxiv.org/html/2608.04646#bib.bib35)\]

## 3Methods

To put the results in context, enable reproducibility, and promote replicability, this section presents an overview of the models considered in the study \(Section[3\.1](https://arxiv.org/html/2608.04646#S3.SS1)\), the psychological tests conducted on the models, including high\-level implementation details, evaluation metrics, and references to the code and data used in the experiments \(Section[3\.2](https://arxiv.org/html/2608.04646#S3.SS2)\), and a discussion on scoring \(Section[3\.3](https://arxiv.org/html/2608.04646#S3.SS3)\)\.

We emphasize that this paper does not provide a controlled causal identification of the effect of RLVR training\. The evaluated systems differ in vendor interfaces, prompting affordances, and access to intermediate reasoning traces, which makes perfectly controlled comparisons difficult\. Accordingly, our contribution should be understood as an empirical behavioral analysis: across a range of ToM tests and prompt perturbations, reasoning\-oriented models appear more stable under variation, a pattern consistent with a robustness\-based interpretation of recent performance gains\.

All prompting classes, raw and graded data, experiment scripts, CLI grading tools, analysis notebooks, and additional detail such as ToM and ToM task definitions, prompting details, and qualitative insight from responses are publicly available at[this GitHub repository](https://github.com/ianbdehaan/Evaluating_ToM_in_Reasoning_Models)\.

### 3\.1Models

Differences in the training of the models and the decisions of their creating companies affect how several properties of models vary\. Some are crucial for the viability and reproducibility of the experiments run\. Those differences are listed in Table[2](https://arxiv.org/html/2608.04646#S3.T2)\.

Table 2:Properties of reasoning models used in the experiments\.Claude is the only model analyzed for which it is possible to turn its thinking off entirely; this allows for experiments with the \(think\) Claude to be reproduced without the thinking, with any differences suggesting the impact that thinking has on its performance, so this is done\. For all the models that allow for temperature configuration \(Table[2](https://arxiv.org/html/2608.04646#S3.T2)\), it is set to 0 to increase reproducibility\. For models that can keep the reasoning in context \(Table[2](https://arxiv.org/html/2608.04646#S3.T2)\), this was done when re\-prompting them\.

Both GPT\-5 and Claude return reasoning summaries that might be filtered by the public interface rather than their full reasoning tokens; however, in preliminary tests, a very big discrepancy in summary qualities was detected, with GPT\-5 often filtering most of its reasoning, while Claude’s summary remains highly useful in understanding its reasoning in all prompts tried\. Due to that, GPT\-5 has to be additionally prompted about the reasoning behind its answers\. Note that comparisons are not perfectly controlled due to API constraints; results should be interpreted as behavioral, not architectural, comparisons\.

### 3\.2Psychological Tests

The psychological tests that are presented in\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\]are applied to reasoning models to assess how well the current reasoning models perform in ToM tasks compared to those reported in the 2023 baseline reference\. This includes classical and modified first and second\-order Sally\-Anne tests, a Strange Stories test, an Imposing Memory test, and modifications on simple ToM task tests\. The latter test was based on the principles behind the prompt modifications shown to make GPT3 fail on simple ToM tasks\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\]\. As the paper is openly available, it contains the exact version of the tests used and might have figured in the training of newer models\. To avoid having models succeed due to their training data, a new task based on each principle was written\. Full details on test and modifications can be found in the supplementary materials in the repository\.

### 3\.3Scoring

Models are evaluated with the same metric for scoring performance as the one used by\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\], with the exception that, for reasoning models that display it, the reasoning is used as the explanation rather than the answer to a follow\-up question\. In detail, 0 points if the reasoning is incorrect, 1 point if the reasoning comes close to the right answer,orif the model subsequently claims they were wrong \(moving from or to a correct answer with proper explanation\), and 2 points if the result and reasoning are correct\. In the results section \([4](https://arxiv.org/html/2608.04646#S4)\), all the reported scores are normalized to the range\[0,1\]\[0,1\]by simply dividing the average result by 2\.

## 4Results

In this section, we present the results from the psychological tests \(Section[4\.1](https://arxiv.org/html/2608.04646#S4.SS1)–[4\.4](https://arxiv.org/html/2608.04646#S4.SS4)\) and we share selected findings from qualitative inspection of model responses to provide insights into both successes and failures in ToM tasks \(Section[4\.5](https://arxiv.org/html/2608.04646#S4.SS5)\)\.

### 4\.11st and 2nd Degree Sally Anne

Table 3:Results obtained for Sally\-Anne \(SA\_1fb\) tests and second\-degree Sally\-Anne \(SA\_2fb\) tests for the different models\. Results are averaged over samples\.As seen in Table[3](https://arxiv.org/html/2608.04646#S4.T3), almost all the answers were correct and had a proper reasoning associated with them\. The only exception is a case where grok\-3\-mini ignores a character’s mental state and merges the knowledge of the two characters when reasoning about the question, merging two characters’ knowledge and, therefore, arriving at the wrong answer\. These results show substantially better performance in these models than the ones verified a few years ago by\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\]\.

Even though for each particular degree of reasoning three different stories are presented, with two being modifications from the original Sally\-Anne test, the good performance in this task alone might still be attributed to the widespread knowledge about this psychological test, as it is the most famous interpretation of the false\-belief test and the general semantic structure of the three versions of the test remain the same, possibly allowing a model to get to the answer through spurious correlations\. The performance in this particular test is too high for any extra analysis\.

### 4\.2Strange Stories

Table 4:Results obtained for the seven types of Strange Stories for different models\. Results are averaged over samples\.The performance of the models in the first five strange stories categories, which involves understanding of, respectively, lies, pretend play, jokes, white lies, and misunderstanding, was flawless \(Table[4](https://arxiv.org/html/2608.04646#S4.T4)\)\. Some models showed slightly more difficulty in the last two categories, sarcasm and double bluff \(Table[4](https://arxiv.org/html/2608.04646#S4.T4)\)\. With sarcasm, both Claude and Grok\-3\-mini made partial errors in the same story\. The story revolves around a father asking his son to clean the kitchen\. The son decides to make an extra effort and also clean the inside of the cabinets, but when he opens it, a pack of flour falls and explodes\. Then, the father gets in the kitchen and says, “Wow, everything is so clean now\!”\. Both models considered, in their reasoning, that the father might have been sarcastic, but ultimately decide he simply looked at the part of the kitchen that was already cleaned and still had not seen the mess, which makes some sense given that the models can’t draw a mental representation of the kitchen to understand it would be almost impossible not to see the mess\.

In the double bluff case, the same thing happens: the three models, in the same story, consider double bluff as a possible explanation in their reasoning, but end up going with another explanation that still makes some logical sense\. The story revolves around a hide\-and\-seek game, where the person playing hide is described as very smart\. The person is trying to avoid being found immediately\. They consider a shed, which offers better concealment, and a tree as the possible hiding spots; they consider that the shed is the obvious choice, but end up going with it anyway\. The models that respond incorrectly consider they might have been double bluffing, but ultimately decide the key is not to be found*immediately*, so the superior concealment of the shed is enough\.

The performance of the models as a whole is close to perfect in the Strange Stories task, indicating how much better models have gotten with time\. Specifically GPT\-5 properly answered and justified all questions\. Interestingly, the two cases in which models partly failed could definitely use a mental visual depiction of a situation to make it clearer\. This will be further discussed in the discussion \(Section[5](https://arxiv.org/html/2608.04646#S5)\)\. Lastly, it is interesting to note that, in this task, Claude, with thinking off performed as well as its reasoning counterpart, but the error pattern is different: Claude is better on sarcasm, worse on double bluff\. It is speculated that the overall high performance is because the strange stories are pretty straightforward, so the possible solution paths are probably not that diverse\.

### 4\.3Imposing Memories

Table 5:Results obtained for the Imposing Memories’ tasks for different models\. Results are averaged over samples\.IntentionalityMemory
The models performed really well in the stories overall, with Claude \(thinking on\) responding correctly to all the questions \(Table[4\.3](https://arxiv.org/html/2608.04646#S4.SS3)\)\. The types of mistakes made vary, but include:

- •Inferring things that are not in the story – i\.e\., he wanted to buy a post for sending a card to his grandmother→\\rightarrowhe has to buy a card for his grandmother
- •Not understanding that the question is about a character’s mental state rather than reality – i\.e\., for the statement ’Hannah: I thought Abi went home sick’ thinking ’Abi went home sick, so it is True’
- •Failing to understand higher degree ToM – i\.e\. for statement ’Hannah: I thought Ama knew that Abi had gone home sick\. Is this true?’ responding ’No, it is not true that Ama knew Abi had gone home sick\(…\)’
- •Hallucinating and responding to something else entirely

The thinking version of Claude performed slightly better than its non\-thinking counterpart \(100% vs 92%\)\. We hypothesize that the difference is more pronounced in this test than in the last two as the inference\-time scaling provided by the reasoning is probably very useful while parsing the story details carefully and comparing them with the claim\.

This task’s difficulty revolves a lot around recovering facts from the histories, which is why it is useful to compare the performance of models in the intentionality questions with the memory ones\. Models made a bit more than twice as many errors in the intentionality class as in the memory one\. However, the errors in the intentionality class were made in the same questions, making the fact that those were hard questions also a valid explanation for the discrepancy\. With this in mind, it is impossible to draw further conclusions from this test\.

### 4\.4Modifications on simple prompts

Table 6:Performance of the Models over modified tasks\.The models we tested perform better on these modifications than GPT\-3 did in\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\], suggesting they are more robust on this specific task than GPT\-3 was \(see supplementary materials in repository for exact definition of modifications\)\. Two of these tasks, 2A and 2B, are nonetheless still particularly difficult for them \(see Table[6](https://arxiv.org/html/2608.04646#S4.T6)\)\. In the next section, we hypothesize that what makes them hard is that the modifications are only trivial when one can picture the scene mentally, and solving them semantically would rely on specifically crafted heuristics\.

Additionally, it’s possible to verify that the reasoning version of Claude performed significantly better in these tests than its non\-thinking counterpart\. It is hypothesized that, with prompts designed to distract models from the optimal reasoning, the model’s reasoning process makes a substantial difference to performance\.

### 4\.5Qualitative Insights from Reasoning Responses

Qualitative inspection of model responses reveals several recurring reasoning patterns that help explain both successes and failures in ToM tasks\.

In many cases, models begin by explicitly filtering and restructuring the narrative into a sequence of factual observations, which appears to help track state changes and identify relevant information for the task\. Reasoning\-oriented models also sometimes perform a perspective\-taking step without explicit prompting, separating what different characters know or believe in a manner similar to perspective\-taking prompting strategies proposed in prior work\. At the same time, reasoning traces occasionally reveal meta\-knowledge about the task itself, such as identifying a scenario as a ‘classic false\-belief test,’ which could in principle serve as a heuristic for solving certain tasks through pattern recognition rather than genuine mental\-state reasoning\. However, it seems unlikely that meta\-knowledge alone accounts for the broader gains, since tasks like the modifications don’t match a known template, though we can’t rule this out quantitatively

Several failure modes also appear in the traces, including indecision between multiple plausible interpretations, confusion between story content and the question being asked, and incorrect reasoning paths that can sometimes be corrected when the model is prompted to reconsider its answer\. Taken together, these qualitative observations suggest that many errors arise from instability in selecting the correct inference path rather than from a complete absence of task\-relevant knowledge, a pattern consistent with our broader interpretation that recent improvements in ToM task performance are associated with increased robustness rather than fundamentally new reasoning capabilities\. For a range of response examples illustrating the above, see the supplementary materials\.

## 5Discussion

In this section, we reflect on the impact of the results given the literature \(Section[5\.1](https://arxiv.org/html/2608.04646#S5.SS1)\) and discuss known limitations and future research \(Section[5\.2](https://arxiv.org/html/2608.04646#S5.SS2)\)\.

### 5\.1Reflection on Results

The results show that models have improved substantially in ToM tasks since 2023\. Part of this is likely just that models have become more capable overall, but in what follows we focus on a second factor: a newfound robustness in their ToM skills\.

Yue et al\.\[[28](https://arxiv.org/html/2608.04646#bib.bib23)\]show that the reasoning paths produced by reasoning models stay bounded by their base models, reaching asymptotic performance without lowering perplexity on the base model, reasoning training does not appear to add fundamentally new capabilities\. Read alongside our own results, this points to a specific interpretation\. In our experiments the thinking\-enabled models, and most clearly the thinking\-on version of Claude compared to its thinking\-off counterpart, were more robust to prompt and task variation\. We therefore hypothesize that the main effect of this kind of reasoning training is improved stability in reaching a solution the model could already, in principle, reach, rather than an expansion of representational capacity\. We stress that our setup compares thinking and non\-thinking models rather than RLVR\-trained and non\-RLVR\-trained ones, so we cannot attribute this effect to RLVR specifically \(see Section[5\.2](https://arxiv.org/html/2608.04646#S5.SS2)\)\.

The psychological tests show just how robust the new models can be: GPT\-5, for instance, made only a single partial mistake across all the tests taken from\[[24](https://arxiv.org/html/2608.04646#bib.bib12)\]\. This robustness seems to matter most in tests specifically designed to disturb a model’s reasoning, which is where the gap between thinking and non\-thinking models was widest, as discussed next\.

Two of the ‘modification’ tasks weren’t successfully completed by a single model\. This may be because those modifications are not as trivial as one might have anticipated, as they might require picturing the scene: for humans, picturing scenes in our minds is so common that it is natural to forget how essential of a skill it can be\. Of course, one can still, through many steps, derive the consequences of transparency semantically for instance, but it becomes a substantially harder task than anticipated\. It is also interesting to notice that one of the only two stories in the ’Strange Stories’ psychological test in which models made a partial mistake had a visual aspect to it that could clarify which of the two hypotheses the models considered in their reasoning is the correct one\.

As seen in Section[2\.2](https://arxiv.org/html/2608.04646#S2.SS2), various prompting strategies have emerged over the years to push LLMs to perform better on ToM tasks\. This suggests models often already had the necessary skills but needed the right prompt to surface them reliably\. Our results point the same way: the models sometimes carry out a version of what perspective\-taking prompting aims to induce without being prompted to do so\.

This is also in line with the findings of\[[21](https://arxiv.org/html/2608.04646#bib.bib29)\], which showed that simple prompt alterations would lead the models to fail in the same types of tasks\. Here we conjecture that this means that the upper bound for success was already present in the models, not that they didn’t possess ’real’ ToM skills\. In other words, they already, to a large extent, had the necessary skills to solve ToM problems, but weird prompts could easily disturb their reasoning\.

To corroborate this, we analyzed the benchmark results from\[[7](https://arxiv.org/html/2608.04646#bib.bib35)\], which proposes a Bayesian theory\-of\-mind framework aligned with our robustness hypothesis\. Although featuring slightly older reasoning models, the results show they consistently outperform their non\-reasoning counterparts on ToM tasks, with even larger gains on the benchmark that most strongly penalizes a lack of robustness\.

With this in mind, the result from\[[28](https://arxiv.org/html/2608.04646#bib.bib23)\]is hereby reinterpreted, at least in the context of ToM tasks\. By understanding what reasoning models don’t do, it is also possible to understand what they do\. More broadly, we hypothesize that the narrowing of the reasoning coverage of RLVR models might be precisely their strength\. While one might want models with very flexible results for some applications, it’s generally desirable for models to be robust\.

### 5\.2Limitations and Further Research

Although evidence from our experiments and benchmarks in literature were found of reasoning helping models in answering queries more robustly in the context of ToM tasks, no clear quantification of such an effect can be provided with only this work\. Future research could tackle such a problem by running ToM benchmark tests comparing RLVR\-trained models with their base model counterparts using the pass@k metric proposed by\[[28](https://arxiv.org/html/2608.04646#bib.bib23)\]\.

Additionally, spurious correlations like relating the semantic structure of the tests with meta\-knowledge of ToM can’t be ruled out for all psychological test cases, which might make some of the results inconclusive for real\-world scenarios\.

The present work also does not directly compare the RLVR\-trained models’ performance in ToM with base models employing other inference\-time scaling techniques; the extent to which the increase in performance can be attributed directly to this specific way of increasing inference time can be investigated in future work\.

Furthermore, some of the evidence motivating the conclusions of this paper is drawn by comparing the performance of Claude with thinking on with its counterpart with thinking off\. Claude, with thinking off is not the same thing as its base model, which is not accessible, yet we hope the difference in performance sheds some light on what the reasoning process is actually helping the model with\. Further research could investigate differences in performance between reasoning models and base models in cases where they are available, such as R1 and V3\-base\. It’s also noteworthy that, although the reasoning traces were analyzed for strategies employed by the models, it has been argued that reasoning traces don’t always present the real motivation behind a model’s final response\[[3](https://arxiv.org/html/2608.04646#bib.bib31)\]\.

Lastly, the different treatment between models might induce differences in performance\. For example, setting the models’ temperature to 0 or asking why a response was given directly affects the performance of the models in the given tests\.

## 6Conclusion

This paper examines the behavior of recent reasoning\-oriented language models on Theory of Mind tasks using a combination of psychological tests, prompt perturbations, qualitative analysis of responses, and benchmark results from prior work\. Across these analyses, reasoning\-oriented models appeared consistently more robust to variations in prompt form and task presentation\. We read this as evidence that recent gains in ToM evaluations reflect improvements in robustness and inference stability, rather than taking it as proof of new ToM\-specific reasoning capabilities, which our behavioral setup cannot directly test\.

More broadly, our results suggest that progress on ToM tasks should not be assessed only through average benchmark performance, but also through robustness under task\-preserving variations\. Given this, some earlier prompt\-sensitive failures may be better understood as failures to reliably reach the correct inference path rather than evidence of a complete absence of task\-relevant competence\. At the same time, our comparisons are behavioral rather than fully controlled, so the conclusions should be interpreted as evidence consistent with a robustness\-based account rather than as a causal attribution to RLVR training\.

Overall, the paper highlights two broader points: first, current models perform substantially better on ToM tasks than earlier generations; second, robustness\-oriented evaluation is essential for interpreting apparent social\-cognitive progress in LLMs\. We hope these findings contribute to a more careful understanding of what reasoning\-oriented models improve, and to the design of future evaluations that better distinguish robustness from underlying capability\.

\{credits\}

#### 6\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]S\. Baron\-Cohen, A\. M\. Leslie, and U\. Frith\(1985\-10\)Does the autistic child have a "theory of mind"?\.Cognition21\(1\),pp\. 37–46\(eng\)\.External Links:ISSN 0010\-0277,[Document](https://dx.doi.org/10.1016/0010-0277%2885%2990022-8)Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p1.1)\.
- \[2\]S\. Baron\-Cohen\(2001\)Theory of mind and autism: a review\.InInternational review of research in mental retardation: Autism \(vol\. 23\), Vol\. 23,pp\. 169–184\.External Links:ISBN 978\-0\-12\-366223\-1Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p1.1)\.
- \[3\]Y\. Chen, J\. Benton, A\. Radhakrishnan, J\. Uesato, C\. Denison, J\. Schulman, A\. Somani, P\. Hase, M\. Wagner, F\. Roger, V\. Mikulik, S\. R\. Bowman, J\. Leike, J\. Kaplan, and E\. Perez\(2025\-05\)Reasoning models don’t always say what they think\.arXiv\.Note:arXiv:2505\.05410 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.05410)Cited by:[§2\.1](https://arxiv.org/html/2608.04646#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2608.04646#S5.SS2.p4.1)\.
- \[4\]Z\. Chen, J\. Wu, J\. Zhou, B\. Wen, G\. Bi, G\. Jiang, Y\. Cao, M\. Hu, Y\. Lai, Z\. Xiong, and M\. Huang\(2024\-08\)ToMBench: benchmarking theory of mind in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15959–15983\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.847)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1)\.
- \[5\]D\. Guo, D\. Yang, H\. Zhang, and et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),ISSN 1476\-4687Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p4.1)\.
- \[6\]J\. Hu, F\. Sosa, and T\. Ullman\(2025\-08\)Re\-evaluating theory of mind evaluation in large language models\.Philosophical Transactions of the Royal Society B: Biological Sciences380\(1932\),pp\. 20230499\.External Links:ISSN 0962\-8436,[Document](https://dx.doi.org/10.1098/rstb.2023.0499),[Link](https://doi.org/10.1098/rstb.2023.0499),https://royalsocietypublishing\.org/rstb/article\-pdf/doi/10\.1098/rstb\.2023\.0499/2821303/rstb\.2023\.0499\.pdfCited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1)\.
- \[7\]H\. Kim, M\. Sclar, T\. Zhi\-Xuan, L\. Ying, S\. Levine, Y\. Liu, J\. B\. Tenenbaum, and Y\. Choi\(2025\)Hypothesis\-driven theory\-of\-mind reasoning for large language models\.InCOLM,Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p6.1),[3rd item](https://arxiv.org/html/2608.04646#S2.I1.i3.p1.1),[§2\.3](https://arxiv.org/html/2608.04646#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2608.04646#S2.T1),[§5\.1](https://arxiv.org/html/2608.04646#S5.SS1.p7.1)\.
- \[8\]H\. Kim, M\. Sclar, X\. Zhou, R\. L\. Bras, G\. Kim, Y\. Choi, and M\. Sap\(2023\-10\)FANToM: a benchmark for stress\-testing machine Theory of Mind in interactions\.arXiv\(en\)\.Note:arXiv:2310\.15421 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.15421)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p6.1)\.
- \[9\]M\. Kosinski\(2024\-11\)Evaluating Large Language Models in theory of mind tasks\.Proceedings of the National Academy of Sciences121\(45\),pp\. e2405460121\.Note:arXiv:2302\.02083 \[cs\]External Links:ISSN 0027\-8424, 1091\-6490,[Document](https://dx.doi.org/10.1073/pnas.2405460121)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1)\.
- \[10\]B\. Meijering, H\. van Rijn, N\. Taatgen, and R\. Verbrugge\(2011\)I do know what you think I think: second\-order theory of mind in strategic games is not that difficult\.InProceedings of the 33th Annual Meeting of the Cognitive Science Society, CogSci 2011, Boston, Massachusetts, USA, July 20\-23, 2011,L\. A\. Carlson, C\. Hölscher, and T\. F\. Shipley \(Eds\.\),Austin,pp\. 2486–2491\.Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p7.1)\.
- \[11\]OpenAIReasoning models \- OpenAI API\.\(en\-US\)\.External Links:[Link](https://platform.openai.com/)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p4.1)\.
- \[12\]D\. Paliotta, J\. Wang, M\. Pagliardini, K\. Y\. Li, A\. Bick, J\. Z\. Kolter, A\. Gu, F\. Fleuret, and T\. Dao\(2025\-02\)Thinking slow, fast: scaling inference compute with distilled reasoners\.arXiv\.Note:arXiv:2502\.20339 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2502.20339)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p4.1)\.
- \[13\]A\. Plaat, M\. van Duijn, N\. van Stein, M\. Preuss, P\. van der Putten, and K\. J\. Batenburg\(2025\-12\)Agentic large language models, a survey\.Journal of Artificial Intelligence Research84\(29\)\.External Links:ISSN 1076\-9757,[Document](https://dx.doi.org/10.1613/jair.1.18675)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.04646#S2.SS1.p1.1)\.
- \[14\]A\. Plaat, A\. Wong, S\. Verberne, J\. Broekens, N\. van Stein, and T\. Bäck\(2025\-12\)Multi\-step reasoning with large language models, a survey\.ACM Comput\. Surv\.58\(6\)\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/3774896)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p1.1),[§1](https://arxiv.org/html/2608.04646#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.04646#S2.SS1.p1.1)\.
- \[15\]F\. Quesque and Y\. Rossetti\(2020\-03\)What do Theory\-of\-Mind tasks actually measure? theory and practice\.Perspectives on Psychological Science15\(2\),pp\. 384–396\(en\)\.External Links:ISSN 1745\-6916, 1745\-6924,[Document](https://dx.doi.org/10.1177/1745691619896607)Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p1.1)\.
- \[16\]I\. Rahwan, M\. Cebrian, N\. Obradovich, J\. Bongard, J\. Bonnefon, C\. Breazeal, J\. W\. Crandall, N\. A\. Christakis, I\. D\. Couzin, M\. O\. Jackson, N\. R\. Jennings, E\. Kamar, I\. M\. Kloumann, H\. Larochelle, D\. Lazer, R\. McElreath, A\. Mislove, D\. C\. Parkes, A\. ’\. Pentland, M\. E\. Roberts, A\. Shariff, J\. B\. Tenenbaum, and M\. Wellman\(2019\-04\)Machine behaviour\.Nature568\(7753\),pp\. 477–486\(en\)\.Note:Publisher: Nature Publishing GroupExternal Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-019-1138-y)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p1.1),[§1](https://arxiv.org/html/2608.04646#S1.p2.1)\.
- \[17\]H\. D\. Schlinger\(2009\-07\)Theory of Mind: an overview and behavioral perspective\.The Psychological Record59\(3\),pp\. 435–448\(en\)\.External Links:ISSN 2163\-3452,[Document](https://dx.doi.org/10.1007/BF03395673)Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p2.1)\.
- \[18\]N\. Shapira, M\. Levy, S\. H\. Alavi, X\. Zhou, Y\. Choi, Y\. Goldberg, M\. Sap, and V\. Shwartz\(2024\-03\)Clever hans or neural Theory of Mind? stress testing social reasoning in Large Language Models\.InProceedings of EACL 2024,Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 2257–2273\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.138)Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p4.1)\.
- \[19\]W\. Street, J\. O\. Siy, G\. Keeling, A\. Baranes, B\. Barnett, M\. McKibben, T\. Kanyere, A\. Lentz, B\. A\. y\. Arcas, and R\. I\. M\. Dunbar\(2026\)LLMs achieve adult human performance on higher\-order theory of mind tasks\.Frontiers in Human NeuroscienceVolume 19 \- 2025\.External Links:[Document](https://dx.doi.org/10.3389/fnhum.2025.1633272),ISSN 1662\-5161Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p3.1)\.
- \[20\]A\. M\. Turing\(1950\-10\)Computing machinery and intelligence\.MindLIX\(236\),pp\. 433–460\.External Links:ISSN 0026\-4423,[Document](https://dx.doi.org/10.1093/mind/LIX.236.433)Cited by:[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p5.1)\.
- \[21\]T\. Ullman\(2023\-03\)Large Language Models fail on trivial alterations to theory\-of\-mind tasks\.arXiv\(en\)\.Note:arXiv:2302\.08399 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2302.08399)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1),[§1](https://arxiv.org/html/2608.04646#S1.p6.1),[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p4.1),[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p5.1),[§3\.2](https://arxiv.org/html/2608.04646#S3.SS2.p1.1),[§4\.4](https://arxiv.org/html/2608.04646#S4.SS4.p1.1),[§5\.1](https://arxiv.org/html/2608.04646#S5.SS1.p6.1)\.
- \[22\]R\. van der Meulen, R\. Verbrugge, and M\. van Duijn\(2025\)Towards properly implementing theory of mind in AI systems: an account of four misconceptions\.External Links:2503\.16468,[Link](https://arxiv.org/abs/2503.16468)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1)\.
- \[23\]B\. van Dijk, T\. Kouwenhoven, M\. Spruit, and M\. J\. van Duijn\(2023\-12\)Large language models: the need for nuance in current debates and a pragmatic perspective on understanding\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12641–12654\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.779)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p1.1)\.
- \[24\]M\. van Duijn, B\. van Dijk, T\. Kouwenhoven, W\. de Valk, M\. Spruit, and P\. van der Putten\(2023\-12\)Theory of mind in large language models: examining performance of 11 state\-of\-the\-art models vs\. children aged 7\-10 on advanced tests\.InProceedings CoNLL 2023,J\. Jiang, D\. Reitter, and S\. Deng \(Eds\.\),Singapore,pp\. 389–402\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.conll-1.25)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p3.1),[§1](https://arxiv.org/html/2608.04646#S1.p6.1),[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.04646#S2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2608.04646#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.04646#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2608.04646#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.04646#S5.SS1.p3.1)\.
- \[25\]A\. Wilf, S\. Lee, P\. P\. Liang, and L\. Morency\(2024\-08\)Think twice: perspective\-taking improves large language models’ theory\-of\-mind capabilities\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8292–8308\.External Links:[Link](https://aclanthology.org/2024.acl-long.451/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.451)Cited by:[1st item](https://arxiv.org/html/2608.04646#S2.I1.i1.p1.1)\.
- \[26\]Y\. Wu, J\. Xie, D\. Zhang, and Z\. Xu\(2025\-05\)DEL\-ToM: inference\-time scaling for Theory\-of\-Mind reasoning via dynamic epistemic logic\.arXiv\(en\)\.Note:arXiv:2505\.17348 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.17348)Cited by:[2nd item](https://arxiv.org/html/2608.04646#S2.I1.i2.p1.1)\.
- \[27\]F\. Xu, Q\. Hao, Z\. Zong, J\. Wang, Y\. Zhang, J\. Wang, X\. Lan, J\. Gong, T\. Ouyang, F\. Meng, C\. Shao, Y\. Yan, Q\. Yang, Y\. Song, S\. Ren, X\. Hu, Y\. Li, J\. Feng, C\. Gao, and Y\. Li\(2025\-01\)Towards large reasoning models: a survey of reinforced reasoning with Large Language Models\.arXiv\(english\)\.Note:arXiv: 2501\.09686 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.09686)Cited by:[§2\.1](https://arxiv.org/html/2608.04646#S2.SS1.p1.1)\.
- \[28\]Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang\(2025\-05\)Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.arXiv\(en\)\.Note:arXiv:2504\.13837 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.13837)Cited by:[§2\.1](https://arxiv.org/html/2608.04646#S2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2608.04646#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2608.04646#S5.SS1.p8.1),[§5\.2](https://arxiv.org/html/2608.04646#S5.SS2.p1.1)\.
- \[29\]W\. X\. Zhao, K\. Zhou, J\. Li, T\. Tang, X\. Wang, Y\. Hou, Y\. Min, B\. Zhang, J\. Zhang, Z\. Dong, Y\. Du, C\. Yang, Y\. Chen, Z\. Chen, J\. Jiang, R\. Ren, Y\. Li, X\. Tang, Z\. Liu, P\. Liu, J\. Nie, and J\. Wen\(2025\-03\)A survey of Large Language Models\.arXiv\(en\)\.Note:arXiv:2303\.18223 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2303.18223)Cited by:[§1](https://arxiv.org/html/2608.04646#S1.p1.1)\.