MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games

# Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games Paper has been submitted to the EMNLP 2026 System Demonstrations track.
Source: [https://arxiv.org/html/2607.10645](https://arxiv.org/html/2607.10645)
###### Abstract

An LLM agent’s public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes\. We presentMafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind\. After every public utterance, every agent privately answers a configurable set of structured probe questions; the answers never re\-enter the game and are scored automatically against the ground truth the engine knows\. An interactive visualizer renders the belief trajectories: impersonate mode shows the game as one agent sees it, panels chart timeline\-aligned accuracy and calibration, and counterfactual replay forks any recorded step\. In a 32\-game DeepSeek case study with 13,815 parsed probe answers, stated confidence is poorly calibrated, with expected calibration error 0\.17, agents over\-predict being suspected 1\.5 times, and a 30\-fork replay experiment walks the counterfactual replay workflow end to end\. Engine, viewer and a corpus of 200\+ cross\-model games are released under an open licence; live demo:[https://karpovilia\.github\.io/mafiascope/](https://karpovilia.github.io/mafiascope/); screencast:[https://vimeo\.com/1208920221](https://vimeo.com/1208920221)\.

MafiaScope: Non\-Invasive, Time\-Resolved Belief Probing for LLM Agents in Social Deduction Games††thanks:Paper has been submitted to the EMNLP 2026 System Demonstrations track\.

Ilia KarpovHSE Universitykarpovilia@gmail\.com

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.10645v1/figures/00-teaser.png)Figure 1:The viewer on an English demo game: ground\-truth graph and log in the centre, side panels are each agent’s private beliefs; node colour = assigned role, edge = suspicion\. The ring on a living Mafioso, “hidNN%”, is the share of the informed crowd missing it\. Okabe\-Ito palette with pattern redundancy\.Multi\-agent LLM systems increasingly operate where success depends on modelling other minds: negotiation, collaboration, multi\-party dialogue\(Tanet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib44); Weiet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib45)\)\. Static Theory\-of\-Mind \(ToM\) benchmarks probe such abilities with self\-contained vignettes or scripted multiparty conversations\(Leet al\.,[2019](https://arxiv.org/html/2607.10645#bib.bib18); Kimet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib20)\), but cannot show how an agent’s model of other agents evolves inside an adversarial interaction it partially causes\(Maet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib57); Riemeret al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib58)\)\. Conversely, work that places LLMs inside social deduction games\(Xuet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib1); O’Gara,[2023](https://arxiv.org/html/2607.10645#bib.bib10)\)typically evaluates only outcomes such as win rates and vote accuracy, leaving the agents’ beliefs a black box, or draws out reasoning inside the game context, where the questioning itself changes subsequent behaviour\. Mafia111[https://en\.wikipedia\.org/wiki/Mafia\_\(party\_game\)](https://en.wikipedia.org/wiki/Mafia_(party_game))distils this challenge: an informed minority, the Mafia, who know each other, covertly kills one player each night; the uninformed majority must unmask them through public discussion and daytime votes\. Success on both sides hinges on modelling, and for the Mafia also managing, what the others believe\.

We demonstrateMafiaScope, a testbed built around one central idea, non\-invasive, time\-resolved belief probing: after every public utterance, privately question every agent with structured questions whose answers never re\-enter the game, and score every answer against the ground truth the engine already knows\.

Measurement is therefore dense in time: every agent is probed after every public message, hundreds of belief snapshots per game, locating belief revision at the level of a single utterance\. It is non\-invasive in a checkable sense: probe answers never enter any agent’s persistent context; §[4](https://arxiv.org/html/2607.10645#S4)spells out this guarantee and its limits\. And it is self\-scoring: Mafia’s hidden roles are known to the engine, so first\-order beliefs \(“Casey is Mafia”\) are directly gradeable, and second\-order beliefs \(“Casey suspects me”\) are gradeable against the target’s own same\-step first\-order reports\.

Contributions\.\(1\) A configurable probe engine with a small yaml DSL \(conditional triggering, probe chaining, per\-probe budgets\) that produces fully indexed JSONL belief logs; \(2\) an interactive visualizer for longitudinal belief inspection with metrics, calibration and deception views, a cross\-game dashboard, and counterfactual replay; \(3\) a 32\-game revised\-instrument case study and 30\-fork replay experiment demonstrating analyses invisible to outcome\-only evaluation, including a second\-order negative result\.

## 2Related Work

LLMs in social deduction games\.Belief tracking over hidden roles is older than LLM agents: DeepRole\(Serrinoet al\.,[2019](https://arxiv.org/html/2607.10645#bib.bib15)\)maintains posterior beliefs over role assignments in Avalon inside a purpose\-built planner; MafiaScope probes and scores beliefs of a closed, off\-the\-shelf LLM\. LLM agents have been studied in Werewolf\(Xuet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib1),[2024b](https://arxiv.org/html/2607.10645#bib.bib2); Bailiset al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib4)\), Avalon\(Lightet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib6); Wanget al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib7)\), Among Us\(Chiet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib8); Golechha and Garriga\-Alonso,[2025](https://arxiv.org/html/2607.10645#bib.bib9)\), Hoodwinked\(O’Gara,[2023](https://arxiv.org/html/2607.10645#bib.bib10)\), and Diplomacy\(Meta Fundamental AI Research Diplomacy Team \(FAIR\)et al\.,[2022](https://arxiv.org/html/2607.10645#bib.bib14)\); Mafia itself has served as a testbed for detecting deceptive actors\(Ibraheemet al\.,[2022](https://arxiv.org/html/2607.10645#bib.bib11); Yoo and Kim,[2024](https://arxiv.org/html/2607.10645#bib.bib13); Costa and Vicente,[2025](https://arxiv.org/html/2607.10645#bib.bib12)\)\. These systems evaluate deception chiefly through outcomes, or draw out reasoning inside the acting context where it shapes subsequent play: recursive contemplation\(Wanget al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib7)\), opponent modelling in card games\(Guoet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib17)\), and MultiMind’s in\-context model of the suspicion each player directs at the agent\(Zhanget al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib16)\), the closest counterpart of our social\-map probe with roles reversed\.Sarkaret al\.\([2025](https://arxiv.org/html/2607.10645#bib.bib55)\)train Among Us agents by multi\-agent RL with each agent’s own per\-message belief about the imposter as a reward signal: per\-message belief readouts serving training, not measurement\. Concurrent work extends the space with per\-statement deception annotation in Werewolf\(Agarwalet al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib59)\), audits of what social\-deduction agents communicate versus internally represent\(Yuanet al\.,[2026](https://arxiv.org/html/2607.10645#bib.bib60)\), and graph\-informed Bayesian belief inference over hidden roles\(Rahimiradet al\.,[2026](https://arxiv.org/html/2607.10645#bib.bib61)\)\. MafiaScope’s object of study is the trajectory of privately probed beliefs, not the win rate\.

Theory\-of\-Mind evaluation\.ToMi\(Leet al\.,[2019](https://arxiv.org/html/2607.10645#bib.bib18)\), FANToM\(Kimet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib20)\), Hi\-ToM\(Wuet al\.,[2023a](https://arxiv.org/html/2607.10645#bib.bib19)\), OpenToM\(Xuet al\.,[2024a](https://arxiv.org/html/2607.10645#bib.bib21)\)and BigToM\(Gandhiet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib22)\)test ToM with static stories, scripted dialogues, or templated vignettes; such competence is brittle\(Ullman,[2023](https://arxiv.org/html/2607.10645#bib.bib24); Sapet al\.,[2022](https://arxiv.org/html/2607.10645#bib.bib25)\)and findings remain contested\(Kosinski,[2024](https://arxiv.org/html/2607.10645#bib.bib26); van Duijnet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib27)\)\. ToMATO\(Shinodaet al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib54)\)is the closest probing design: role\-playing LLMs verbalize their mental state at every utterance of generated conversations, but the result is frozen into a static QA benchmark\. In interactive settings, InterIntent\(Liuet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib35)\)grades intention understanding inside Avalon games, and SOTOPIA\(Zhouet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib36)\)scores social goal completion with an LLM judge; MafiaScope instead scores privately probed beliefs against engine ground truth, repeating the same probe after every utterance of a game the agent itself shapes, with stakes and information asymmetry arising naturally\.

Belief probing and self\-report\.Whether language models hold beliefs at all is debated\(Haseet al\.,[2021](https://arxiv.org/html/2607.10645#bib.bib33)\); probing beliefs by asking is imperfect\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.10645#bib.bib28); Turpinet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib29)\), and whether models can introspect is an active question\(Binderet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib30); Lindsey,[2025](https://arxiv.org/html/2607.10645#bib.bib31)\)\. Activation\-level probing decodes belief states of self and others directly from hidden representations\(Zhuet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib56)\); MafiaScope stays behavioural by design, so the instrument also applies to closed API models whose activations are out of reach\. We treat probe answers not as philosophical belief representations\(Herrmann and Levinstein,[2025](https://arxiv.org/html/2607.10645#bib.bib32)\)but as operational belief reports with a checkable predictive structure \(§[7](https://arxiv.org/html/2607.10645#S7)\), and the engine records raw and parsed answers so faithfulness itself can be studied\.

Multi\-agent observation interfaces\.Observation interfaces exist for generative\-agent societies\(Parket al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib37)\)and multi\-agent frameworks\(Wuet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib40); Chenet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib38); Wuet al\.,[2023b](https://arxiv.org/html/2607.10645#bib.bib39)\); recent HCI work adds debugging and goal\-tracking views over agent conversations\(Eppersonet al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib41); Cosciaet al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib43)\)\. Table[1](https://arxiv.org/html/2607.10645#S2.T1)contrasts the capabilities most relevant to belief measurement across released systems whose tooling we could verify from public documentation\. Individual ingredients thus exist separately in prior work; what we have not found elsewhere is their combination in one released instrument: out\-of\-band probing guarantee, per\-utterance density, automatic scoring against engine ground truth, interactive viewer, and counterfactual replay\.

Table 1:Interface capabilities of MafiaScope \(MS\) vs\. ChatArena \(CA\)\(Wuet al\.,[2023b](https://arxiv.org/html/2607.10645#bib.bib39)\), AutoGen with AGDebugger \(AG\)\(Wuet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib40); Eppersonet al\.,[2025](https://arxiv.org/html/2607.10645#bib.bib41)\), Werewolf Arena \(WA\)\(Bailiset al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib4)\), SOTOPIA \(ST\)\(Zhouet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib36)\), and InterIntent \(II\)\(Liuet al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib35)\), assessed from each system’s paper, documentation and repository\. Parenthesized marks are partial: WA exposes in\-context private reasoning without out\-of\-band probing or scoring; II scores intention guessing against players’ self\-stated intentions, but in\-context without viewer \(logs only\); AG edits and resets messages and re\-runs from an intermediate state, in that dimension strictly stronger than our reroll\-only forking, but yields single re\-runs, notNN\-fold outcome distributions; ST shows per\-episode judge scores, not timeline\-aligned metrics\.
## 3System Overview

MafiaScope consists of three decoupled layers connected by gt logs \(Figure[1](https://arxiv.org/html/2607.10645#S1.F1)shows the viewer\):

1. 1\.Game engine\(game\.py\): a classic Mafia loop \(night kill / doctor save→\\today discussion→\\topublic vote\) fornnplayers with roles Mafia, Doctor, Villager \(the engine also implements a Sheriff, absent from the corpus\)\. Players are LLM agents parameterized by backend, language \(English/Russian prompt packs\) and a Big\-Five personality vector rendered into the system prompt\(Jianget al\.,[2024](https://arxiv.org/html/2607.10645#bib.bib48); Serapio\-Garcíaet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib49)\)\. Backends form a named registry referenced per player slot, so one game can field a heterogeneous line\-up \(ChatGPT Mafia against a local Qwen town\): OpenAI\-compatible APIs \(hosted or local vLLM/Ollama\), local HuggingFace models \(with a batched GPU backend\), or a plain\-HTTP message bus through which an external process in any language plays seats; a bundled Arena panel assigns backends to seats and launches batches\. Every public event carries a composite key\(round, public\_msg\_seq\); every probe record extends it with\(player\_idx, probe\_seq\), so the exact information state behind any answer is reconstructible\.
2. 2\.Introspection probe engine\(introspection\.py\): described in §[4](https://arxiv.org/html/2607.10645#S4)\.
3. 3\.Data pipeline and visualizer:prepare\_viewer\.pyjoins game and probe logs into per\-step viewer states;serve\_viewer\.pyserves a single\-page D3 application\.

## 4The Probe Engine

After each public utterance and private night action, and once per round, the engine pauses the game and privately interrogates each alive agent\. Each probe runs on a throw\-away copy of the agent’s message list: question and answer live only for the duration of the probe pass and are never written back, so probe text never enters any game\-facing LLM call\. For stateless API backends the probed game is therefore distributed identically to an unprobed one, up to provider\-side effects we cannot observe; for local backends, probe calls sample under a forked RNG state and are never batched with game calls, though batch scheduling may still differ\.

Probes are declared in YAML:

\-id:social\_map

question:\>

Youare\{player\}\.Hereisyourcurrent

roleassessment:\{prev\_role\_assessment\}

Estimatehoweachaliveplayer

\(\{players\}\)feelsaboutYOU\.\.\.

JSON:\{"toward\_me":\[\.\.\.\]\}

The DSL supportsconditional triggering\(when: own\_turnvs\. after every message\),probe chaining\(\{prev\_<id\>\}/\{last\_<id\>\}splice an earlier probe’s parsed, re\-serialized answer into a later question; raw\-text fallbacks are flagged\), andper\-probe token budgets\. Answers are parsed as JSON, tolerant of markdown fences and truncation; both raw and parsed forms are logged\.

Six probes ship by default:role\_beliefsandrole\_assessment\(first\-order role attributions with confidence and rationale\),suspicion\_ranking,planned\_action\(post\-turn intent\),social\_map\(second\-order: each player’s attitude toward me\), andpersonality\_profile\(Big\-Five attribution, gradeable against the target’s generating vector\); the case\-study corpus runs five of these \(all butrole\_beliefs\)\.

Measurement density is paid for in calls: in the case\-study corpus probing adds 631 calls per game on top of 27 game\-move calls, multiplying LLM traffic by roughly 24 at∼\\sim540 tokens per probe call\. The probe set is a configuration knob: the replay experiment runs a suspicion\-only set at a fifth of this cost\.

## 5The Visualizer

The viewer \(Figures[1](https://arxiv.org/html/2607.10645#S1.F1)and[2](https://arxiv.org/html/2607.10645#S5.F2)\) is a dependency\-light single\-page application \(vanilla JS \+ D3\): ground\-truth graph, one agent’s subjective graph, and the filtered game log over a shared timeline; hovering a belief edge shows the agent’s verbatim rationale\.

![Refer to caption](https://arxiv.org/html/2607.10645v1/figures/08-impersonate.png)Figure 2:Impersonate mode: the game as Mafia agent Logan experiences it\. Logan’s world: his first\-order beliefs\. Believed: who Logan thinks trusts/suspects him\. Actual: what those players privately reported about Logan at the same timestep; the gap between panels is second\-order error, scored live \(match 2/4; Mafia partner excluded\)\.Impersonate mode is the system’s centrepiece: it re\-renders the entire interface from inside one agent, filtering the log to what the agent can observe and the graph to its own beliefs, and contrasting two ego\-panels: its second\-order expectation \(“believed”\) against the others’ actual same\-step first\-order reports about it \(“actual”\), scored live in the panel headers; Mafia\-Mafia pairs are excluded, a partner’s certainty reflects role knowledge\. Deception becomes visible: a successful Mafioso’s “actual” panel keeps reporting trust while the ground\-truth panel shows Mafia\.

![Refer to caption](https://arxiv.org/html/2607.10645v1/figures/14-metrics-strip.png)Figure 3:The in\-viewer metrics panel, aligned with the timeline: per\-agent first\-order accuracy, crowd Mafia\-detection recall with per\-Mafioso deception success \(dashed\), and per\-agent second\-order consistency \(scored as in §[7](https://arxiv.org/html/2607.10645#S7)\)\. Hovering shows values; clicking seeks the timeline\.In\-viewer analytics\.The viewer computes the paper’s evaluation measures on the fly \(Figure[3](https://arxiv.org/html/2607.10645#S5.F3)\): a metrics panel plots each agent’s first\-order accuracy, the crowd’s Mafia\-detection recall, and second\-order consistency over the shared timeline; a deception overlay rings each living Mafioso with its current deception success; a calibration view \(Figure[4](https://arxiv.org/html/2607.10645#A3.F4)\) bins stated confidence against actual accuracy per agent or corpus\-wide; and a companion dashboard aggregates the same statistics across all recorded games\.

Researcher workflow\.A typical session: skim the dashboard for an anomalous game; open it and scrub the timeline to the step where crowd recall collapses; impersonate the agent that misled the vote, inspecting its private beliefs and second\-order gap; then branch just before the suspect utterance and replay repeatedly \(§[6](https://arxiv.org/html/2607.10645#S6)\)\. Every view is addressable by URL fragment, so analytical states can be cited and scripted; all data are plain JSONL, so any panel can be recomputed outside the viewer\. One English demo game,36594b66, deliberately serves as the running example, chosen for its legible pile\-on; that outcome\-based selection is one more reason the replay numbers are illustrative\.

Scalability\.The viewer is demonstrated at the shipped corpora’s scale: seven agents, a few dozen timeline steps\. The limit is legibility, not computation: a subjective panel draws a belief edge for every pair of players, trivial to render but increasingly hard to read\. Focus\-plus\-context strategies already help: impersonate mode, per\-agent log filtering, suspicion\-threshold edge rendering, cross\-game aggregation; we expect readability up to roughly a dozen agents, beyond which per\-agent filtering becomes mandatory\.

## 6Counterfactual Replay

Dense probing localizes the utterance at which a belief flipped, but localization alone is correlational; counterfactual replay \(“branch from here”\) closes this gap\. With snapshotting enabled, the engine serializes every agent’s full message context at every step and can restore any composite key from the snapshot, re\-simulating the continuationNNtimes into an outcome distribution\. Forks are driven from the viewer’s Branch button \(server\-side, Figure[5](https://arxiv.org/html/2607.10645#A5.F5)\) or from the command line for scripted batches; branches arrive in the game list as ordinary games, render as a branch tree, and can themselves be forked; forking at every step profiles when the outcome was still open\.

Pivotal\-utterance attribution\.BranchingNNtimes directly before a candidate utterance \(PRE arm: the speaker resamples it\) and directly after it \(POST arm: the utterance is fixed\) turns “the beliefs moved here” into an estimated arm contrast\. Intervention editing is not yet implemented, so PRE replaces the utterance with a draw from the speaker’s own counterfactual utterance distribution: the strongest design reroll\-only forking supports\. We illustrate the workflow on the running\-example game, in which villager Gray was eliminated 5:1 after a Mafia deflection \(R1\.7\), a Mafia\-partner pile\-on \(R1\.8\) and a villager endorsement \(R1\.9\): 30 forks \(3 utterances×\\times2 arms×\\times5 rerolls, suspicion\-only probes\) yield point estimates favouring the pile\-on: fixing R1\.8 raisesP​\(eliminated=Gray\)P\(\\mathrm\{eliminated\}\{=\}\\mathrm\{Gray\}\)from 0\.4 to 0\.8, versus\+0\.2\+0\.2for R1\.7 and0\.00\.0for R1\.9, with matching vote\-time suspicion shifts \(Appendix[E](https://arxiv.org/html/2607.10645#A5)\)\. Statistically, the largest contrast is 2/5 vs\. 4/5 eliminations \(Fisher exact, two\-sided,p≈0\.52p\\approx 0\.52\) atn=5n\{=\}5per arm, with no multiplicity correction across utterances and outcomes: the experiment suggests, but does not establish, pivotality; its purpose is demonstrating the counterfactual workflow end to end\.P​\(Mafia​wins\)P\(\\mathrm\{Mafia\\ wins\}\)stayed at 0\.8\-1\.0 in every arm, too few rerolls to bound how local the effect is\. A planned intervention editor \(utterance rewriting, night\-action overrides, belief injection\) will turn the correlational vote coupling of F4 into causal tests\.

## 7Case Study

The case\-study corpus is 32 seven\-player games, 2 Mafia, 1 Doctor, 4 Villagers, recorded with the revised instrument: clean social\-map wording, parsed\-answer chaining, 960\-token budgets, per\-step snapshots; 21 games run the English prompt pack, 11 the Russian; of 20,199 probe calls, 13,815 returned parsed answers; Mafia won 31 of 32 versus 17 of 30 in the legacy corpus, the earlier pre\-revision batch\. The shift is not a language artefact: Mafia won 21/21 EN and 10/11 RU games here; drift behind the unpinned alias and campaign differences remain candidates\. The findings grade beliefs, not wins\. The purpose of the case study is not to establish new properties of DeepSeek, but to demonstrate the kinds of analyses MafiaScope enables\. 95% CIs come from a cluster bootstrap resampling whole games,B=1000B\{=\}1000, fixed seed\. All findings describedeepseek\-chatunder this configuration, not LLM agents in general; each finding exercises one capability: trajectories \(F1\), calibration \(F2\), second\-order scoring \(F3\), vote coupling \(F4\), budgets \(F5\)\. How fast users localize errors, and whether replay helps, is what the formative study designed in Appendix[D](https://arxiv.org/html/2607.10645#A4)will measure; it has not yet run\.

F1: Belief trajectories are measurable\.Villager\-side agents start largely agnostic \(74\.9% “Unknown” in round 0\) and commit increasingly accurate beliefs as evidence accumulates: committed accuracy rises from 47\.6% to 58\.4% by round 2 and 75\.4% in the small round\-3 cell, against chance rates of 41\-45% and with a round\-1 dip to chance; recall of the true Mafia rises from 5\.4% to 60\.9% by round 2 \(CI \[44\.0, 75\.0\]\)\. Per\-round numbers and two instrument confounds are in Appendix[B](https://arxiv.org/html/2607.10645#A2): the trajectory is a joint property of agent and instrument\.

F2: Agents’ confidence is poorly calibrated\.Stated confidence carries little signal below its top bin: accuracy stays between 43\.0% and 46\.0% from confidence 40 to 79, and reaches only 54\.6% at 80\-99, thirty points below that bin’s mean confidence \(per\-bin counts in Appendix[C](https://arxiv.org/html/2607.10645#A3)\)\. Over fixed\-width confidence bins of width 20 \(confidence 100 folded into the top bin\), ECE is 0\.168 \(CI \[0\.130, 0\.203\]\) and the Brier score 0\.283\(Guoet al\.,[2017](https://arxiv.org/html/2607.10645#bib.bib53)\); in the typology ofMoore and Healy \([2008](https://arxiv.org/html/2607.10645#bib.bib52)\)this is overprecision, not overestimation; better\-calibrated verbalized confidence can be prompted for\(Tianet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib34)\)\.

F3: Agents over\-predict being suspected\.Each social\-map prediction \(“BBsuspects me”\) is graded againstBB’s own same\-step role assessment of the predictor; the canonical rule is in Appendix[C](https://arxiv.org/html/2607.10645#A3)\. These agents predict “suspects” 1\.53 times as often as suspicion actually occurs \(CI \[1\.44, 1\.64\];n=15,638n\{=\}15\{,\}638pairs\), a machine analogue of the human spotlight effect\(Gilovichet al\.,[2000](https://arxiv.org/html/2607.10645#bib.bib51)\), present under both template wordings and in both corpus languages; scoring caveats, threshold sensitivity and the wording ablation are in Appendix[C](https://arxiv.org/html/2607.10645#A3)\. Split by the predictor’s true role, the effect belongs to the innocent: non\-Mafia predictors over\-predict by 1\.84 \(CI \[1\.67, 2\.04\]\) while Mafia predictors are nearly calibrated \(1\.08, CI \[0\.94, 1\.25\]\), which speaks against rational vigilance of the guilty; the split is observational, Mafia’s probe input differs\.

F4: Probed beliefs track votes\.Counting only probes taken before the voter’s own vote, an innocent agent’s day vote lands on the top suspect of its latest suspicion ranking in 64\.9% of votes \(n=74n\{=\}74, CI \[54\.0, 75\.7\]; chance 27\.3% = a uniform vote over alive others\) and inside its committed\-Mafia set in 71\.2% \(n=80n\{=\}80, CI \[62\.7, 79\.2\]; chance 41\.7% = set size over alive others\)\. Mafia follows its stated ranking as often \(70\.0%,n=30n\{=\}30\), but its committed\-set alignment stays at chance \(29\.6% vs 25\.2%,n=27n\{=\}27, CI \[14\.3, 46\.7\]\): it will not vote where its private set points, at its partner\. Probes taken after the vote match it almost perfectly \(95\.1% top\-1\), so reports absorb the agent’s own actions in real time\. The coupling is correlational, but probe reports track behaviour, not idle text\.

F5: Probe budgets shape the measurement\.Under the legacy corpus’s 400\-token cap, 73% of role\-assessment answers truncated mid\-JSON, and a JSON\-repair pass had to recover all 4,238 unparsed answers \(its mechanics: Appendix[B](https://arxiv.org/html/2607.10645#A2)\)\. At the present corpus’s 960\-token budgets truncation disappears\.

Belief dynamics as a temporal graph\.The probe logs also read as a continuous\-time directed multigraph: nodes are agents; every probe answer emits timestamped, confidence\-weighted edges: suspicion, trust and neutral attitudes, role guesses, about 631 belief events per game\. A companion script turns visible thrashing between targets into metrics over this stream: suspicion volatility \(meanL1L\_\{1\}shift between consecutive suspicion vectors, renormalized on common support so deaths do not inflate it\) and top\-suspect flip rate \(share of consecutive probes whose top suspect changes, death\-forced flips excluded\)\. Over the case\-study corpus, volatility averages 0\.300 \(CI \[0\.284, 0\.318\]\), the top suspect changes in 48\.7% of consecutive probe pairs \(CI \[45\.1, 52\.8\],n=2,583n\{=\}2\{,\}583\), and 51\.6% of flips return to a previously abandoned suspect \(CI \[44\.9, 57\.1\]\): agents circle rather than converge\. The legacy corpus’s Mafia\-instability asymmetry, 0\.197 vs\. 0\.146 volatility, does not replicate here: 0\.312 vs\. 0\.294 with overlapping CIs\. The script also exports the edge stream in the event format of temporal graph networks\(Rossiet al\.,[2020](https://arxiv.org/html/2607.10645#bib.bib62); Xuet al\.,[2020](https://arxiv.org/html/2607.10645#bib.bib63); Kazemiet al\.,[2020](https://arxiv.org/html/2607.10645#bib.bib64)\); we release corpus and metrics as an enabler for temporal\-graph modelling; training such models is future work, and we claim no modelling results\.

## 8Audience, Licence, Availability

MafiaScope targets researchers of machine ToM and multi\-agent LLM behaviour; engine, probes, viewer, and dataset are MIT\-licensed\. The dataset carries the 32\-game case\-study corpus, pinned by a machine\-readable manifest in the repository: 5 English demo games behind the screenshots, 5 clean\-wording Russian, 22 release\-generation; plus the old\-wording ablation arm, the legacy 30\-game batch, and the 30 replay forks of §[6](https://arxiv.org/html/2607.10645#S6)\. The full release spans 200\+ cross\-model games\.

## 9Limitations and Roadmap

Probe answers are self\-reports and may be unfaithful\(Turpinet al\.,[2023](https://arxiv.org/html/2607.10645#bib.bib29)\), though F4 ties innocents’ reports to their votes; raw generations are logged for audit; test\-retest reliability under nonzero temperature is unquantified\. The replay experiment is reroll\-only,N=5N\{=\}5per arm, on a deliberately chosen game\. The viewer has had no user evaluation; the protocol of Appendix[D](https://arxiv.org/html/2607.10645#A4)is designed, not yet run\.

## Ethics and Broader Impact

The testbed studies deception in a fictional, consented game frame among artificial agents; no human subjects are involved, and no personal data is processed\. Research into how LLMs deceive and detect deception is dual\-use: the same insights that harden systems against manipulation could inform manipulative applications\. We release measurement tooling \(probes, scoring, visualization\) rather than optimized deception policies, and the dataset contains only synthetic game dialogue\. The replay facility could in principle be repurposed as an optimization loop for persuasive utterances; we ship it as an attribution instrument\. ToM vocabulary \(“beliefs”, “suspects”\) is used operationally, as defined by the probes\.

## References

- M\. Agarwal, S\. Rana, T\. Sundoro, H\. Berhe, S\. Kim, V\. Sharma, S\. O’Brien, and K\. Zhu \(2025\)WOLF: werewolf\-based observations for LLM deception and falsehoods\.Note:arXiv:2512\.09187; MTI\-LLM Workshop @ NeurIPS 2025Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- S\. Bailis, J\. Friedhoff, and F\. Chen \(2024\)Werewolf arena: a case study in LLM evaluation via social deduction\.External Links:2407\.13943,[Link](https://arxiv.org/abs/2407.13943)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- F\. J\. Binder, J\. Chua, T\. Korbak, H\. Sleight, J\. Hughes, R\. Long, E\. Perez, M\. Turpin, and O\. Evans \(2024\)Looking inward: language models can learn about themselves by introspection\.arXiv preprint arXiv:2410\.13787\.External Links:[Link](https://arxiv.org/abs/2410.13787)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.
- W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\. Chan, H\. Yu, Y\. Lu, Y\. Hung, C\. Qian, Y\. Qin, X\. Cong, R\. Xie, Z\. Liu, M\. Sun, and J\. Zhou \(2024\)AgentVerse: facilitating multi\-agent collaboration and exploring emergent behaviors\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2308.10848)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- Y\. Chi, L\. Mao, and Z\. Tang \(2024\)AmongAgents: evaluating large language models in the interactive text\-based social deduction game\.Note:Wordplay Workshop @ ACL 2024External Links:2407\.16521,[Link](https://arxiv.org/abs/2407.16521)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- A\. Coscia, S\. Guo, E\. Koh, and A\. Endert \(2025\)OnGoal: tracking and visualizing conversational goals in multi\-turn dialogue with large language models\.InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology \(UIST ’25\),External Links:[Document](https://dx.doi.org/10.1145/3746059.3747746),[Link](https://arxiv.org/abs/2508.21061)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- D\. B\. Costa and R\. Vicente \(2025\)Deceive, detect, and disclose: large language models play mini\-mafia\.External Links:2509\.23023,[Link](https://arxiv.org/abs/2509.23023)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- W\. Epperson, G\. Bansal, V\. Dibia, A\. Fourney, J\. Gerrits, E\. Zhu, and S\. Amershi \(2025\)Interactive debugging and steering of multi\-agent AI systems\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems \(CHI ’25\),Yokohama, Japan\.External Links:[Document](https://dx.doi.org/10.1145/3706598.3713581),[Link](https://arxiv.org/abs/2503.02068)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- K\. Gandhi, J\. Fränken, T\. Gerstenberg, and N\. D\. Goodman \(2023\)Understanding social reasoning in language models with language models\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\), Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2306.15448)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- T\. Gilovich, V\. H\. Medvec, and K\. Savitsky \(2000\)The spotlight effect in social judgment: an egocentric bias in estimates of the salience of one’s own actions and appearance\.Journal of Personality and Social Psychology78\(2\),pp\. 211–222\.External Links:[Document](https://dx.doi.org/10.1037/0022-3514.78.2.211)Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p4.3)\.
- S\. Golechha and A\. Garriga\-Alonso \(2025\)Among us: a sandbox for measuring and detecting agentic deception\.External Links:2504\.04072,[Link](https://arxiv.org/abs/2504.04072)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p3.1)\.
- J\. Guo, B\. Yang, P\. Yoo, B\. Y\. Lin, Y\. Iwasawa, and Y\. Matsuo \(2023\)Suspicion\-agent: playing imperfect information games with theory of mind aware GPT\-4\.External Links:2309\.17277,[Link](https://arxiv.org/abs/2309.17277)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- P\. Hase, M\. Diab, A\. Celikyilmaz, X\. Li, Z\. Kozareva, V\. Stoyanov, M\. Bansal, and S\. Iyer \(2021\)Do language models have beliefs? Methods for detecting, updating, and visualizing model beliefs\.arXiv preprint arXiv:2111\.13654\.External Links:[Link](https://arxiv.org/abs/2111.13654)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.
- D\. A\. Herrmann and B\. A\. Levinstein \(2025\)Standards for belief representations in LLMs\.Minds and Machines35\(1\)\.External Links:[Document](https://dx.doi.org/10.1007/s11023-024-09709-6),[Link](https://arxiv.org/abs/2405.21030)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.
- S\. Ibraheem, G\. Zhou, and J\. DeNero \(2022\)Putting the con in context: identifying deceptive actors in the game of mafia\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 158–168\.External Links:[Link](https://aclanthology.org/2022.naacl-main.11/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.11)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- H\. Jiang, X\. Zhang, X\. Cao, C\. Breazeal, D\. Roy, and J\. Kabbara \(2024\)PersonaLLM: investigating the ability of large language models to express personality traits\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 3605–3627\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.229/)Cited by:[item 1](https://arxiv.org/html/2607.10645#S3.I1.i1.p1.3)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.External Links:[Link](https://arxiv.org/abs/2207.05221)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.
- S\. M\. Kazemi, R\. Goel, K\. Jain, I\. Kobyzev, A\. Sethi, P\. Forsyth, and P\. Poupart \(2020\)Representation learning for dynamic graphs: a survey\.Journal of Machine Learning Research21\(70\),pp\. 1–73\.Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p7.2)\.
- H\. Kim, M\. Sclar, X\. Zhou, R\. Le Bras, G\. Kim, Y\. Choi, and M\. Sap \(2023\)FANToM: a benchmark for stress\-testing machine theory of mind in interactions\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 14397–14413\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.890/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.890)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1),[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- M\. Kosinski \(2024\)Evaluating large language models in theory of mind tasks\.Proceedings of the National Academy of Sciences121\(45\),pp\. e2405460121\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2405460121),[Link](https://www.pnas.org/doi/10.1073/pnas.2405460121)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- M\. Le, Y\. Boureau, and M\. Nickel \(2019\)Revisiting the evaluation of theory of mind through question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 5872–5877\.External Links:[Link](https://aclanthology.org/D19-1598/),[Document](https://dx.doi.org/10.18653/v1/D19-1598)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1),[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- J\. Light, M\. Cai, S\. Shen, and Z\. Hu \(2023\)AvalonBench: evaluating LLMs playing the game of avalon\.External Links:2310\.05036,[Link](https://arxiv.org/abs/2310.05036)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- J\. Lindsey \(2025\)Emergent introspective awareness in large language models\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2025/introspection/index.html)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.
- Z\. Liu, A\. Anand, P\. Zhou, J\. Huang, and J\. Zhao \(2024\)InterIntent: investigating social intelligence of LLMs via intention understanding in an interactive game context\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 6718–6746\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.383/)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- Z\. Ma, J\. Sansom, R\. Peng, and J\. Chai \(2023\)Towards a holistic landscape of situated theory of mind in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 1011–1031\.Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1)\.
- Meta Fundamental AI Research Diplomacy Team \(FAIR\), A\. Bakhtin, N\. Brown, E\. Dinan, G\. Farina, C\. Flaherty, D\. Fried, A\. Goff, J\. Gray, H\. Hu, A\. P\. Jacob, M\. Komeili, K\. Konath, M\. Kwon, A\. Lerer, M\. Lewis, A\. H\. Miller, S\. Mitts, A\. Renduchintala, S\. Roller, D\. Rowe, W\. Shi, J\. Spisak, A\. Wei, D\. Wu, H\. Zhang, and M\. Zijlstra \(2022\)Human\-level play in the game of diplomacy by combining language models with strategic reasoning\.Science378\(6624\),pp\. 1067–1074\.External Links:[Document](https://dx.doi.org/10.1126/science.ade9097)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- D\. A\. Moore and P\. J\. Healy \(2008\)The trouble with overconfidence\.Psychological Review115\(2\),pp\. 502–517\.External Links:[Document](https://dx.doi.org/10.1037/0033-295X.115.2.502)Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p3.1)\.
- A\. O’Gara \(2023\)Hoodwinked: deception and cooperation in a text\-based game for language models\.External Links:2308\.01404,[Link](https://arxiv.org/abs/2308.01404)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1),[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology \(UIST ’23\),San Francisco, CA, USA\.External Links:[Document](https://dx.doi.org/10.1145/3586183.3606763),[Link](https://arxiv.org/abs/2304.03442)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- S\. Rahimirad, G\. Gergerli, L\. Romero, A\. Qian, M\. L\. Olson, S\. Stepputtis, and J\. Campbell \(2026\)Bayesian social deduction with graph\-informed language models\.Note:arXiv:2506\.17788; accepted to ACL 2026 main conferenceCited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- M\. Riemer, Z\. Ashktorab, D\. Bouneffouf, P\. Das, M\. Liu, J\. D\. Weisz, and M\. Campbell \(2025\)Position: theory of mind benchmarks are broken for large language models\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\), Position Paper Track,Note:arXiv:2412\.19726Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1)\.
- E\. Rossi, B\. Chamberlain, F\. Frasca, D\. Eynard, F\. Monti, and M\. Bronstein \(2020\)Temporal graph networks for deep learning on dynamic graphs\.Note:arXiv:2006\.10637; ICML 2020 Workshop on Graph Representation LearningCited by:[§7](https://arxiv.org/html/2607.10645#S7.p7.2)\.
- M\. Sap, R\. Le Bras, D\. Fried, and Y\. Choi \(2022\)Neural theory\-of\-mind? on the limits of social intelligence in large LMs\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 3762–3780\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.248/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.248)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- B\. Sarkar, W\. Xia, C\. K\. Liu, and D\. Sadigh \(2025\)Training language models for social deduction with multi\-agent reinforcement learning\.InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),Note:arXiv:2502\.06060Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- G\. Serapio\-García, M\. Safdari, C\. Crépy, L\. Sun, S\. Fitz, P\. Romero, M\. Abdulhai, A\. Faust, and M\. Matarić \(2023\)Personality traits in large language models\.arXiv preprint arXiv:2307\.00184\.External Links:[Link](https://arxiv.org/abs/2307.00184)Cited by:[item 1](https://arxiv.org/html/2607.10645#S3.I1.i1.p1.3)\.
- J\. Serrino, M\. Kleiman\-Weiner, D\. C\. Parkes, and J\. B\. Tenenbaum \(2019\)Finding friend and foe in multi\-agent games\.InAdvances in Neural Information Processing Systems 32 \(NeurIPS 2019\),pp\. 1249–1259\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/912d2b1c7b2826caf99687388d2e8f7c-Abstract.html)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- K\. Shinoda, N\. Hojo, K\. Nishida, S\. Mizuno, K\. Suzuki, R\. Masumura, H\. Sugiyama, and K\. Saito \(2025\)ToMATO: verbalizing the mental states of role\-playing LLMs for benchmarking theory of mind\.InProceedings of the AAAI Conference on Artificial Intelligence,Note:arXiv:2501\.08838Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- C\. Tan, J\. Gu, and Z\. Ling \(2023\)Is ChatGPT a good multi\-party conversation solver?\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 4905–4915\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.326),[Link](https://aclanthology.org/2023.findings-emnlp.326/)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330),[Link](https://aclanthology.org/2023.emnlp-main.330/)Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p3.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1),[§9](https://arxiv.org/html/2607.10645#S9.p1.1)\.
- T\. Ullman \(2023\)Large language models fail on trivial alterations to theory\-of\-mind tasks\.arXiv preprint arXiv:2302\.08399\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2302.08399),[Link](https://arxiv.org/abs/2302.08399)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- M\. van Duijn, B\. van Dijk, T\. Kouwenhoven, W\. de Valk, M\. Spruit, and P\. van der Putten \(2023\)Theory of mind in large language models: examining performance of 11 state\-of\-the\-art models vs\. children aged 7\-10 on advanced tests\.InProceedings of the 27th Conference on Computational Natural Language Learning \(CoNLL\),J\. Jiang, D\. Reitter, and S\. Deng \(Eds\.\),Singapore,pp\. 389–402\.External Links:[Link](https://aclanthology.org/2023.conll-1.25/),[Document](https://dx.doi.org/10.18653/v1/2023.conll-1.25)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- S\. Wang, C\. Liu, Z\. Zheng, S\. Qi, S\. Chen, Q\. Yang, A\. Zhao, C\. Wang, S\. Song, and G\. Huang \(2023\)Avalon’s game of thoughts: battle against deception through recursive contemplation\.External Links:2310\.01320,[Link](https://arxiv.org/abs/2310.01320)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- J\. Wei, K\. Shuster, A\. Szlam, J\. Weston, J\. Urbanek, and M\. Komeili \(2023\)Multi\-party chat: conversational agents in group settings with humans and models\.arXiv preprint arXiv:2304\.13835\.External Links:[Link](https://arxiv.org/abs/2304.13835)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2024\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.InFirst Conference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2308.08155)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- Y\. Wu, Y\. He, Y\. Jia, R\. Mihalcea, Y\. Chen, and N\. Deng \(2023a\)Hi\-ToM: a benchmark for evaluating higher\-order theory of mind reasoning in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 10691–10706\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.717/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.717)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- Y\. Wu, Z\. Jiang, A\. Khan, Y\. Fu, L\. Ruis, E\. Grefenstette, and T\. Rocktäschel \(2023b\)ChatArena: multi\-agent language game environments for large language models\.Note:GitHub repositoryExternal Links:[Link](https://github.com/Farama-Foundation/chatarena)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p4.1)\.
- D\. Xu, C\. Ruan, E\. Korpeoglu, S\. Kumar, and K\. Achan \(2020\)Inductive representation learning on temporal graphs\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§7](https://arxiv.org/html/2607.10645#S7.p7.2)\.
- H\. Xu, R\. Zhao, L\. Zhu, J\. Du, and Y\. He \(2024a\)OpenToM: a comprehensive benchmark for evaluating theory\-of\-mind reasoning capabilities of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8593–8623\.External Links:[Link](https://aclanthology.org/2024.acl-long.466/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.466)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- Y\. Xu, S\. Wang, P\. Li, F\. Luo, X\. Wang, W\. Liu, and Y\. Liu \(2023\)Exploring large language models for communication games: an empirical study on werewolf\.External Links:2309\.04658,[Link](https://arxiv.org/abs/2309.04658)Cited by:[§1](https://arxiv.org/html/2607.10645#S1.p1.1),[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- Z\. Xu, C\. Yu, F\. Fang, Y\. Wang, and Y\. Wu \(2024b\)Language agents with reinforcement learning for strategic play in the werewolf game\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.External Links:[Link](https://proceedings.mlr.press/v235/xu24ad.html)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- B\. Yoo and K\. Kim \(2024\)Finding deceivers in social context with large language models and how to find them: the case of the mafia game\.Scientific Reports14,pp\. 30946\.External Links:[Document](https://dx.doi.org/10.1038/s41598-024-81997-5),[Link](https://www.nature.com/articles/s41598-024-81997-5)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- Y\. Yuan, R\. Song, W\. Li, Z\. Li, H\. Liu, X\. Kong, C\. Han, Y\. Yang, Z\. Zhao, Z\. Dong, F\. Lyu, B\. He, H\. Wu, J\. Kang, and X\. Liu \(2026\)QUACK: questioning, understanding, and auditing communicated knowledge in multimodal social deduction agents\.Note:arXiv:2605\.27068Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- Z\. Zhang, N\. Xiao, Q\. Chai, D\. Ye, and H\. Wang \(2025\)MultiMind: enhancing werewolf agents with multimodal reasoning and theory of mind\.InProceedings of the 33rd ACM International Conference on Multimedia,External Links:[Document](https://dx.doi.org/10.1145/3746027.3755752),[Link](https://arxiv.org/abs/2504.18039)Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p1.1)\.
- X\. Zhou, H\. Zhu, L\. Mathur, R\. Zhang, H\. Yu, Z\. Qi, L\. Morency, Y\. Bisk, D\. Fried, G\. Neubig, and M\. Sap \(2024\)SOTOPIA: interactive evaluation for social intelligence in language agents\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2310.11667)Cited by:[Table 1](https://arxiv.org/html/2607.10645#S2.T1),[§2](https://arxiv.org/html/2607.10645#S2.p2.1)\.
- W\. Zhu, Z\. Zhang, and Y\. Wang \(2024\)Language models represent beliefs of self and others\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,pp\. 62638–62681\.Note:arXiv:2402\.18496Cited by:[§2](https://arxiv.org/html/2607.10645#S2.p3.1)\.

## Appendix ADefault Probe Set

The six default probes with full question templates, trigger conditions and token budgets ship inconfigs/config\.yaml; the case\-study configurations areconfig\_deepseek\.yaml\(Russian pack, five probes\) andconfig\_en\_demo\.yaml\(English pack\), the old\-wording ablationconfig\_ablation\_demand\.yaml\. Probe chaining order isrole\_assessment→\\tosocial\_map\. The case\-study agents call the DeepSeek API aliasdeepseek\-chat, accessed July 2026, provider\-default sampling, no temperature override; replay forks and the old\-wording ablation arm are excluded from the corpus\. The API does not expose the served model version, so the pin we can offer is the alias plus per\-game access timestamps recorded in each trace’s setup event\.

## Appendix BFirst\-Order Trajectories and Repair

Table 2:First\-order beliefs of villager\-side agents by round \(32 games, repaired answers\): committed\-guess accuracy vs\. chance \(permutation of the alive\-role multiset\), “Unknown” abstention share, and recall of true Mafia \(abstentions = misses\) vs\. chance \(alive Mafia over alive others\)\. Games end when a side wins; shrinkingnnreflects survivorship, and the round\-3 sample is small \(recall 93\.5%, CI \[50\.0, 100\.0\]\)\.Two artefacts inflate the trajectory in Table[2](https://arxiv.org/html/2607.10645#A2.T2)independently of any real belief improvement\. First, probe chaining re\-injects the agent’s previous assessment into the next question, an anchoring ratchet that can amplify commitment independently of evidence\. Second, game mechanics help: eliminations shrink the candidate pool and publicly reveal roles, and survivorship removes the trajectories of eliminated players\.

JSON repair keeps only the complete prefix of a truncated answer and writes nothing itself; on the present corpus it recovers zero answers\. The 10 games with near\-zero probe loss reproduce every headline number within the full\-corpus CIs\.

## Appendix CCalibration Bins and Second\-Order Scoring Sensitivity

![Refer to caption](https://arxiv.org/html/2607.10645v1/figures/15-calibration.png)Figure 4:The calibration view for a single game: stated confidence bins of fresh role\-assessment guesses against their actual accuracy \(dashed diagonal = perfect calibration; hatched = overconfident; whiskers = Wilson 95% intervals; bins withn<10n<10dimmed\)\. The corpus\-level plateau below confidence 80 \(F2\) is visible in individual games as well\.Corpus\-level calibration bins for F2: accuracy 46\.4% at confidence 20\-39 \(n=360n\{=\}360\), 43\.0% at 40\-59 \(n=2,320n\{=\}2\{,\}320\), 46\.0% at 60\-79 \(n=3,338n\{=\}3\{,\}338\), 54\.6% at 80\-99 \(n=994n\{=\}994\); the 0\-19 bin \(n=4n\{=\}4\) is too small to interpret\. Both language arms are miscalibrated in the same direction: ECE 0\.150 \(CI \[0\.109, 0\.184\]\) in the 21 English games, 0\.189 \(CI \[0\.119, 0\.256\]\) in the 11 Russian ones\.

A natural objection to F3: the legacy social\-map template itself contained a suggestive sentence, “if you suspect someone, they may sense it”, so the probe could have implanted the very belief it measures\. To test this we recorded two matched Russian batches of 5 games differing only in that sentence: the over\-prediction appears under both wordings, 1\.58 \(CI \[1\.46, 1\.77\]\) with the sentence and 1\.48 \(CI \[1\.39, 1\.61\]\) without it, and in the English batch as well \(1\.45, CI \[1\.18, 1\.72\]\)\. So the effect is not an artefact of the wording; whether the sentence added a little on top cannot be judged from 5 games per wording, where the intervals overlap\. The case\-study corpus uses the clean wording throughout\.

The F3 measure is a consistency between two self\-reports, not ToM against engine ground truth\. Under the canonical rule, agreement exceeds a trivial majority baseline on this corpus: 54\.7% versus 44\.7% \(\+10\.1\+10\.1pp, CI\[5\.7,12\.4\]\[5\.7,12\.4\]\); on the legacy pre\-revision corpus it fell below the same baseline \(−4\.3\-4\.3pp\)\. The ratio itself is stable across the grid: 1\.32 \[1\.21, 1\.41\] at threshold 30, 1\.53 at 50, 3\.05 \[2\.59, 3\.82\] at 70, smallest at the most liberal threshold, so no compressed\-denominator artefact\. The split survives an input control: on pairs with a confident own assessment \(70\+, matching Mafia’s knowing input\) innocents over\-predict at 1\.95 \[1\.73, 2\.21\] vs Mafia 1\.24 \[1\.08, 1\.48\], the confident Mafia ratio edging above 1\. That gap remains threshold\-dependent: Table[3](https://arxiv.org/html/2607.10645#A3.T3)reports the F3 comparison \(32 games, repaired answers, Mafia\-Mafia pairs excluded\) over the confidence threshold for mapping a role guess to an attitude\. The majority class is “trusts” at threshold 30 and “neutral” at 50 and 70\.

Table 3:F3 sensitivity grid: agreement vs\. majority baseline under different confidence thresholds \(n=15,638n\{=\}15\{,\}638pairs throughout; the staleness window is inert on this corpus and omitted\)\. Agreement beats the baseline at the canonical threshold 50, matches it at 30 and falls below it at 70: the comparison remains threshold\-dependent\.
## Appendix DFormative User Study Protocol

Within\-subject,N=5N\{=\}5researchers external to the project: two error\-localization tasks on unfamiliar recorded games, one with the viewer and one on raw JSONL logs with scripting allowed; tool order and task\-tool assignment are counterbalanced, 12 minutes per task\. Measures: time to localization, correctness against gold labels, and SUS with three open questions on viewer affordances\. The full session script, task games with gold answers, and questionnaires ship indocs/user\_study\_protocol\.md\.

## Appendix EReplay Experiment Details

![Refer to caption](https://arxiv.org/html/2607.10645v1/figures/16-branch-dialog.png)Figure 5:The viewer’s branch dialog: any timeline step of a snapshotted game can be forked intoNNreplays server\-side; branches arrive in the game list as ordinary games and render as a branch tree\.Per\-utterance arm contrasts for the experiment of §[6](https://arxiv.org/html/2607.10645#S6)\(the running\-example game36594b66from the case\-study corpus;n=5n=5rerolls per arm; suspicion\-only probes; the POST arm’s within\-arm variance serves as the resampling noise floor; vote\-time suspicion is the mean normalized rank of the player oversuspicion\_rankingprobes in the fork round’s vote phase, 1 = most suspicious\)\. In the factual game Gray was eliminated with vote\-time suspicion 0\.96 and Mafia won in round 2\.

None of the contrasts in Table[4](https://arxiv.org/html/2607.10645#A5.T4)is statistically significant at these sample sizes: the largest, R1\.8’s 2/5 vs\. 4/5 eliminations, has Fisher exact \(two\-sided\)p≈0\.52p\\approx 0\.52, no interval on the arm difference excludes zero, and 3 utterances×\\times4 outcome measures are inspected without multiplicity correction\. Comparisons of pivotality between utterances additionally carry a fork\-position confound: a later fork leaves less stochasticity before the vote, so PRE probabilities of utterances at different timeline positions are not aligned\. The table therefore documents an end\-to\-end run of the attribution workflow; it suggests, but does not establish, R1\.8’s pivotality\.

Table 4:Reroll\-variance attribution over three candidate utterances of the round\-1 discussion: Logan’s deflection onto Gray \(R1\.7\), Finley’s pile\-on \(R1\.8\), Casey’s endorsement \(R1\.9\)\.Δ\\Delta= POST−\-PRE\. All 30 forks completed and ship as replayable logged games\.