Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

# Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Source: [https://arxiv.org/html/2608.18083](https://arxiv.org/html/2608.18083)
Micha Heilbron2,3

1IDEAS Research Institute; Warsaw, Poland 2Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands 3University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands karolina\.drozdz@ideas\.edu\.pl,micha\.heilbron@mpi\.nl

###### Abstract

Understanding language requires tracking entities across discourse – i\.e\., knowing where things are and how they change, even when not explicitly stated\. Whether language models perform such tracking in a human\-like fashion remains unclear, in part because existing evaluations rely on artificial tasks, far removed from natural language comprehension, and lack comparisons to humans\. Here, we evaluate entity tracking in both language models and humans \(N=48N=48\) using naturalistic narratives at multiple levels of complexity\. In humans, we find that entity tracking degrades specifically with narrative complexity, not narrative length\. In language models, we find that human\-level entity tracking is already present at 410 million parameters – well below the multi\-billion parameter, code\-specialised models identified by prior work – and improves with scale, with contemporary models far exceeding human performance\. Together, these results demonstrate that entity tracking, a core component of language understanding, emerges at model scales far smaller than previously thought\.

Entity tracking emerges in sub\-billion parameter language models and exceeds human performance in naturalistic narratives

Karolina Drożdż1and Micha Heilbron2,31IDEAS Research Institute; Warsaw, Poland2Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands3University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlandskarolina\.drozdz@ideas\.edu\.pl,micha\.heilbron@mpi\.nl

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/abstract.png)Figure 1:Conceptual overview\. Left: language understanding requires*tracking entities*in an internal model of discourse context, as a narrative unfolds\. Right: in humans \(top\), entity tracking accuracy decreases with situational complexity; in language models \(bottom\), we find a similar pattern, and an increase in capability with scale\.## 1Introduction

The fluency of large language models \(LLMs\) has reignited a long\-standing question: does the capacity to produce coherent text imply genuine language understanding, or can it be mimicked through sophisticated pattern matching\(Bender and Koller,[2020](https://arxiv.org/html/2608.18083#bib.bib11); Shanahan,[2024](https://arxiv.org/html/2608.18083#bib.bib12)\)? One way to operationalise this question is to ask whether language models \(LMs\)*track entities*across discourse as human comprehenders do – that is, maintaining and updating representations of the underlying states, locations, and relations of referents as a narrative unfolds\(Groenendijk and Stokhof,[1991](https://arxiv.org/html/2608.18083#bib.bib13); Heim,[2002](https://arxiv.org/html/2608.18083#bib.bib14); Zwaan and Radvansky,[1998](https://arxiv.org/html/2608.18083#bib.bib22)\)\. Critically, many of these states are never explicitly mentioned but must be inferred from described events, based on a coherent internal model of the text’s meaning\. For instance, if a story describes a key being removed from a box containing only a key, comprehenders immediately infer that the box is now empty, even though this is never mentioned\(Liet al\.,[2021](https://arxiv.org/html/2608.18083#bib.bib17)\)\. To what extent do language models that exhibit apparent understanding actually track entities, like a genuine comprehender?

Prior work has begun to address this question from several angles\. First, influential probing work showed that entity states could be linearly decoded from encoder\-decoder model activations\(Liet al\.,[2021](https://arxiv.org/html/2608.18083#bib.bib17)\), though subsequent work revealed that most of the reported accuracy derived from trivial cases rather than genuine entity tracking\(Kim and Schuster,[2023](https://arxiv.org/html/2608.18083#bib.bib18)\)\. Analysing model outputs,Kim and Schuster \([2023](https://arxiv.org/html/2608.18083#bib.bib18)\); Kimet al\.\([2024](https://arxiv.org/html/2608.18083#bib.bib20)\)found that robust entity tracking surfaced only in large models pretrained on massive, code\-specialised datasets\.

However, some limitations complicate the conclusions that can be drawn from this body of work\. First, no study has compared model performance to human performance, despite evidence that humans routinely rely on shallow, “good\-enough” strategies rather than building fully detailed situation models\(Ferreiraet al\.,[2002](https://arxiv.org/html/2608.18083#bib.bib10)\)\. The extent to which humans reliably track entities across narratives is thus an empirical question, that matters for interpreting model performance\. Second, the tasks used in prior evaluations are highly formulaic and artificial – e\.g\., “Box 0 contains the painting, Box 1 contains the bell, Box 2 contains the guitar, … Move the glass from Box 6 to Box 4\. …”\(Kim and Schuster,[2023](https://arxiv.org/html/2608.18083#bib.bib18); Kimet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib20)\)– making these tasks closer to solving a reasoning puzzle or executing a program than comprehending natural language\. The parameter and data requirements reported by these studies may thus reflect demands of the task, rather than limits on entity tracking itself\.

In this work, we address both limitations\. We developed a controlled experimental paradigm in which short narratives – programmatically generated but naturalistic and readable – describe scenes at five levels of situational complexity\. Entity tracking is assessed via both explicit \(generation\-based\) and implicit \(forced\-choice or probability read\-out\) methods, enabling comparison between humans and models under matched conditions, with different task demandsHu and Frank \([2024](https://arxiv.org/html/2608.18083#bib.bib9)\)\. We evaluate two fully open model families across scales – Pythia \(70M–12B\) and OLMo 2 \(1B–32B\) – alongside larger, contemporary models \(Llama 3\.3, Qwen 2\.5\), and compare their performance to humans \(N=48N=48\) on the same stimuli\. To assess whether model performance reflects genuine tracking or reliance on distributional priors, we additionally test all models on narratives containing pseudowords and semantically improbable objects\.

We report four findings\. First, human entity tracking degrades specifically with situational complexity, revealing a cognitive complexity cost to maintaining structured situation models\. Second, entity tracking in LMs emerges at far smaller scales than previously reported: robust, human\-level tracking is present at 410 million parameters already, and the ability improves predictably with scale, with the effect of complexity diminishing as models grow, and completely disappearing for contemporary models at 70B scale\. Third, instruction tuning selectively improves explicit but not implicit entity tracking, indicating that the underlying representational capacity is present in base models prior to alignment\. Fourth, performance is robust across standard, pseudoword, and semantically improbable object types, indicating that models robustly track entities through discourse structure rather than lexical association\.

Together, these results demonstrate that entity tracking – a core component of genuine language comprehension – emerges with model scale, at parameter counts far below those previously associated with this ability, and that at sufficient scale language models can far exceed human performance\.

Beyond the empirical findings, we share our evaluation paradigm, enabling researchers to study naturalistic entity tracking in small, fully open base models without task\-specific fine\-tuning111Code is available in our[repository](https://anonymous.4open.science/r/entity_tracking_EMNLP-A842)\.\.

## 2Related Work

Whether generative models build structured internal representations of the domains they are trained on – so\-called “world models” – has been studied extensively in artificial domains where the ground truth is fully specified, such as games or planetary motion\. Overall, evidence for such representations has been inconsistent across domains\. On the one hand, models trained on Othello move sequences develop internal board\-state representations that support predictions even on unreachable game states\(Liet al\.,[2022](https://arxiv.org/html/2608.18083#bib.bib16); Nandaet al\.,[2023](https://arxiv.org/html/2608.18083#bib.bib1)\)\. On the other hand, Transformers trained on urban navigation harboured spatially incoherent internal maps despite near\-perfect predictionVafaet al\.\([2024](https://arxiv.org/html/2608.18083#bib.bib15)\); and models trained on planetary trajectories failed to recover the true, underlying Newtonian mechanicsVafaet al\.\([2025](https://arxiv.org/html/2608.18083#bib.bib6)\)\. Together, in such closed domains, perfect next\-token prediction does not imply perfect internal models\.

For natural language, the question of whether language models construct coherent ‘situation models’ and dynamically track entity states has been approached at the representational and mechanistic level\.Prakashet al\.\([2024](https://arxiv.org/html/2608.18083#bib.bib19)\), focusing on a simplified version of the entity tracking task without any state changes, identified circuits implementing entity tracking in base language models and showed that fine\-tuning enhances these pathways\. Studying Transformers that were specifically trained on permutation compositionLiet al\.\([2025](https://arxiv.org/html/2608.18083#bib.bib7)\)identified structured parallel algorithms \(associative scans\) for entity tracking\. This establishes that the computational machinery exists in Transformers, but not whether this is used in language comprehension or if it emerges from natural language pretraining\.

The most direct prior assessment comes fromKim and Schuster \([2023](https://arxiv.org/html/2608.18083#bib.bib18)\), who used the previously described boxes\-and\-objects task and found that pure\-text models up to 175 billion parameters failed entirely; tracking first surfaced only in GPT\-3\.5, a closed, code\-specialised model\.Kimet al\.\([2024](https://arxiv.org/html/2608.18083#bib.bib20)\)extended this to open\-weight model families, locating robust tracking at 13B parameters with 500 billion additional code tokens, or at 7B only under far more extensive code training \(2 trillion tokens\)\. However, both evaluate models on this artificial, programmatic task and lack comparison to humans – leaving open whether their findings generalise to the kind of situation\-model construction that characterises natural narrative comprehension\. Our work addresses both limitations\.

## 3Methods

### 3\.1Task Design

We developed a novel entity tracking task in which participants and models track the locations of objects as they move across locations within short narratives of varying complexity\. Each narrative is followed by a question that probed the final location of a target object \(see[AppendixD](https://arxiv.org/html/2608.18083#A4)for narrative examples\)\.

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/methods.png)Figure 2:Overview of the experimental design\. The top panel displays a simplified narrative at complexity 3, with two target objects \(F0, F1\)\. The bottom panels show the two evaluation formats: anexplicitopen recall task \(left\) requiring generation of the correct final location of the evaluated object, and animplicittask \(right\) administered as a forced\-choice selection for human participants and as a probability read\-out comparing correct versus incorrect continuations for language models\. For full example narratives at multiple complexitiy levels, see[AppendixD\.](https://arxiv.org/html/2608.18083#A4)Complexity\.Complexity was defined by the number of objects, locations, and movements to be tracked\. Five levels were constructed: C1 involved a single object in one location; C2 a single object moving between two locations; C3 two objects across three locations; C4 three objects with interleaved movements across three locations; and C5 four objects across four locations\. Human participants were tested on C1–C4, as pilot data indicated that C4 already imposed substantial cognitive load; LMs were additionally tested on C5\.

Evaluation formats\.Because explicit question\-answering can systematically underestimate model capabilities\(Hu and Frank,[2024](https://arxiv.org/html/2608.18083#bib.bib9)\), we used both explicit and implicit evaluation formats\. For models, theexplicitcondition required free generation, while theimplicitcondition utilised direct probability read\-outs\. Correspondingly, for human participants, theexplicitandimplicitconditions consisted of open\-ended generation and forced\-choice selection, respectively \(see[Figure2](https://arxiv.org/html/2608.18083#S3.F2)\)\.

Recency control\.Since narratives of higher complexity are also longer, this introduces a potential confound: accuracy decreases could reflect recency effects rather than complexityper se\. To address this, we varied the position of the target object within longer narratives \(i\.e\., focus object, F0–F3\)\. This allowed us to assess whether performance depended on how recently an object was mentioned, and thus to separate recency\-based recall from the demands of tracking multiple entities\. If performance were driven by recency, more recently mentioned objects should yield higher accuracy; conversely, stable performance across positions would indicate reliance on integrated representations rather than surface\-level recall\.

### 3\.2Stimuli

All narratives were generated using an automated pipeline\. This enabled a systematic and scalable construction of novel stimuli that are guaranteed to not appear verbatim in the training data\. The resulting narratives were designed to be naturalistic rather than rigid or artificial, following a fixed event structure to maintain coherence\. Objects, locations, and characters were randomly sampled for each instance, ensuring diversity across trials\.

To discourage participants from adopting a strategic search for object locations – rather than constructing a coherent situation model – we introduced minor lexical variation \(e\.g\., descriptive adjectives in∼\\sim30% of sentences\) and interleaved 10 filler trials with structurally distinct narratives and comprehension questions targeting general story details\.

Template variants\.To assess reliance on semantic priors, we constructed three template conditions: \(i\)standard, using objects commonly found in the described context; \(ii\)pseudoword, using pronounceable nonwords generated with WuggyKeuleers and Brysbaert \([2010](https://arxiv.org/html/2608.18083#bib.bib33)\)matched in form; and \(iii\)improbable, using semantically incongruent or abstract entitiesWanget al\.\([2018](https://arxiv.org/html/2608.18083#bib.bib32)\)\(see[AppendixB](https://arxiv.org/html/2608.18083#A2)\)\. Since the confound of exploiting training data distributions is unique to LMs, they require adversarial evaluations to demonstrate robust tracking\. Therefore, while the human baseline was established using only the standard template, models were evaluated across all three templates to test whether their performance reflects a generalisable mechanism or merely semantic plausibility\.

### 3\.3Human Evaluation

Sample\.We recruitedN=48N=48native English speakers via Prolific and compensated them for their participation \(mean age = 40\.8,S​D=12\.9SD=12\.9; 24 male, 20 female, 4 not disclosed\)\. The data collection protocol was approved by the Ethics Review Board of the Faculty of Social and Behavioural Sciences at the University of Amsterdam \(decision number: FMG\-8670\), and participants provided informed consent\. Participants were required to complete the full study and confirm attentiveness and absence of AI assistance\. Participants with accuracy below 50% on a given task were excluded from that task’s analysis\. This yielded final samples ofN=41N=41for the explicit task andN=43N=43for the implicit task\.

Procedure\.Participants completed 46 trials \(36 main, 10 filler\)\. The main trials were evenly split between explicit and implicit tasks \(18 trials each\) and balanced across seven configurations \(8×\\timesC1\.F0, 8×\\timesC2\.F0, 4×\\timesC3\.F0, 4×\\timesC3\.F1, 4×\\timesC4\.F0, 4×\\timesC4\.F1, 4×\\timesC4\.F2\)\. Narratives were presented on a single screen, followed by a probe\. Reading time was limited to 40 seconds and response time was limited to 15 seconds \(including reading the question\)\. These constraints were designed to target entity tracking as it naturally occurs during comprehension, rather than treating the task as a reasoning puzzle\. Importantly, it also greatly reduced the feasibility of external aids such as note\-taking or querying LLMs\.

Data Analysis\.We fit a logistic generalised linear model with trial\-level binary accuracy as the dependent variable\. Predictors included mean\-centred complexity, probing method, and their interaction, with standard errors clustered by participant\. Recency was included as a covariate\.

### 3\.4Model Evaluation

Model Selection\.We evaluated two fully open model families: Pythia\(Bidermanet al\.,[2023](https://arxiv.org/html/2608.18083#bib.bib23)\)and OLMo 2\(Team OLMo,[2025](https://arxiv.org/html/2608.18083#bib.bib24)\)\. These models provide full access to training data and checkpoints, enabling controlled scaling analyses\. For the Pythia suite, we tested all available sizes: 70M, 160M, 410M, 1B, 1\.4B, 2\.8B, 6\.9B, and 12B parameters\. OLMo 2 models included 1B, 7B, 13B, and 32B variants \(base and instruction\-tuned\)\. We additionally evaluated two large contemporary models, Llama 3\.3 \(70B\)Llama Team \([2024](https://arxiv.org/html/2608.18083#bib.bib35)\)and Qwen 2\.5 \(72B\)Qwen Team \([2024](https://arxiv.org/html/2608.18083#bib.bib36)\), as high\-performing reference models\.

Evaluation protocol\.Each model was evaluated on 50 trials per condition \(complexity×\\timesobject position\), yielding 550 trials per task\. For implicit trials, we computed summed log\-probabilities of candidate continuations scoring a response as correct if the consistent continuation received higher probability\. For explicit trials, models were prompted \(the exact prompt used can be found in[AppendixC](https://arxiv.org/html/2608.18083#A3)\) to produce a single\-word location response\. Responses were considered correct if they contained the target location term, allowing minor formatting variation\. Base models were evaluated primarily on the implicit task, as they are not optimised for instruction\-following\.

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/figA.png)Figure 3:Humans show declining accuracy with increasing complexity while contemporary 70B LMs \(Llama 3\.3 and Qwen 2\.5\) maintain ceiling performance\. \(A\) Explicit entity tracking task\. \(B\) Implicit entity tracking task\. Error bars represent 95% confidence intervals\.Data Analysis\.Models were treated as individual case studies\. Accuracy was computed across conditions, with comparisons focusing on systematic variation across model size, instruction tuning, and template type\. Human–model comparisons were conducted descriptively by contrasting the distribution of human accuracy with the performance of individual models under matched conditions\.

## 4Results

### 4\.1Human entity tracking performance declines with narrative complexity, independent of recency

We first asked whether humans reliably track entities at various complexity levels\. Participants performed well above chance, at 75\.3% accuracy on the explicit and 79\.3% on the implicit task\. Notably, performance declined with increasing narrative complexity: in the explicit task, accuracy decreased from 87\.2% at C1 to 65\.0% at C4; in the implicit task, from 89\.0% to 76\.0% \(b=−0\.38b=\-0\.38,SE=0\.07\\text\{SE\}=0\.07,z=−5\.72z=\-5\.72,p<0\.001p<0\.001;[Figure 3](https://arxiv.org/html/2608.18083#S3.F3)\); each unit increase in complexity reduced the odds of a correct response by approximately 32% \(OR=0\.68\\text\{OR\}=0\.68, 95% CI\[0\.60,0\.78\]\[0\.60,0\.78\]\), and this effect was statistically indistinguishable across explicit and implicit tasks \(b=−0\.09b=\-0\.09,SE=0\.05\\text\{SE\}=0\.05,z=−1\.73z=\-1\.73,p=0\.08p=0\.08\)\.

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/figB.png)Figure 4:Model scaling effects on implicit entity tracking accuracy\. \(A\) Pythia models \(70M to 12B parameters\)\. \(B\) OLMo 2 base models \(1B to 32B parameters\)\. Larger models generally show improved performance, with the exception of OLMo 32B\-base\.This complexity effect could, in principle, reflect recency or narrative length, since higher\-complexity narratives are also longer\. To control for this, our design varied the relevant object’s position within longer narratives, allowing us to isolate the effect of recency at matched complexity\. Including target object’s position as a covariate, the complexity effect remained significant \(b=−0\.53b=\-0\.53,S​E=0\.10SE=0\.10,z=−5\.45z=\-5\.45,p<\.001p<\.001;O​R=0\.59OR=0\.59, 95% CI 0\.48, 0\.71\), while recency was not a significant predictor \(b=0\.19b=0\.19,S​E=0\.11SE=0\.11,z=1\.63z=1\.63,p=\.10p=\.10\)\. Human performance thus appears primarily limited by the complexity of the situation model itself — the number of entities and relations being tracked — not by surface\-level recency\.

### 4\.2Contemporary models maintain ceiling performance across complexity levels

We then compared human performance against two contemporary 70B\-class models, Llama 3\.3 and Qwen 2\.5\. While human accuracy fell steeply with complexity, both models held near\-ceiling accuracy across all levels \(see[Figure 3](https://arxiv.org/html/2608.18083#S3.F3)\)\. This model class thus operates far beyond human cognitive capacity, with no trace of the complexity cost that limits human comprehenders\.

### 4\.3Entity tracking scales with model size

Because contemporary models saturate the task, we turned to fully open model suites at smaller scales, to trace where this representational capacity emerges, in both base and instruction\-tuned models\. We evaluated Pythia \(70M to 12B parameters\) and OLMo 2 \(1B to 32B parameters\) on the same stimuli \(see[Figure 4](https://arxiv.org/html/2608.18083#S4.F4)\), on both the implicit \(forced\-choice\) and explicit \(generation\) tasks\.

On the implicit task, all base models showed substantial entity tracking and exhibited a human\-like effect of complexity: performance declined as situational complexity increased \(see[Figure 4](https://arxiv.org/html/2608.18083#S4.F4)\)\. Critically, for both model families, overall performance increased with scale\. Pythia accuracy improved monotonically from 53\.5% at 70M to 89\.6% at 12B\. OLMo 2 showed a similar but higher\-baseline trajectory, with smaller complexity costs throughout \(e\.g\., 18 points at 1B; 7 points at 13B\)\.

One exception to the overall scaling trend was OLMo 2 32B, which underperformed the 13B model\. We believe this reflects the outlying nature of this particular model – potentially related to documented modifications to the training pipeline of the 32B modelAllen Institute for AI \([2025](https://arxiv.org/html/2608.18083#bib.bib38)\)– rather than a genuine reversal of the scaling trend\.

On the explicit task, instruction\-tuned OLMo 2 models – which were explicitly trained for question answering – showed a similar scaling pattern, improving from 55\.3% at 1B to 93\.5% at 32B \([Figure 5](https://arxiv.org/html/2608.18083#S4.F5)A;[AppendixA](https://arxiv.org/html/2608.18083#A1)\)\. Here, the 32B model was again an outlier, but now in a positive sense, far outperforming the 13B \(93\.5% vs 65\.6%\)\. Pythia has no instruction\-tuned variant, and base models predictably performed at or below chance on the explicit task; thus, we did not analyze their explicit task performance further\.

Together, these results indicate that entity tracking is present in base language models below 1 billion parameters, improves predictably with scale, and – like in humans – is modulated by situational complexity\.

### 4\.4Instruction tuning improves explicit but not implicit performance

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/figD.png)Figure 5:Instruction tuning effects on OLMo 2 models \(A\) Explicit task\. \(B\) Implicit task\. Instruction tuning substantially improves explicit task performance, with gains increasing with model size\. In contrast, instruction tuning has mixed effects on implicit task performance\.![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/figC.png)Figure 6:\(A\) Examples of standard, pseudoword, and improbable object types used in the stimuli\. \(B\) Explicit task accuracy\. \(C\) Implicit task accuracy\. Object type does not affect model performance\.To examine the effect of instruction tuning, we compared OLMo 2 base models with their instruction\-tuned counterparts \(see[Figure 5](https://arxiv.org/html/2608.18083#S4.F5)\)\. Instruction tuning substantially improved explicit task performance – with gains up to \+52\.4 percentage points – consistent with the task’s demand for open\-ended question answering\. In contrast, instruction tuning did not improve implicit performance, and in some cases slightly reduced it \(e\.g\., OLMo 2 1B: 88\.5% to 78\.0%\)\. This indicates instruction tuning enhances the ability to express entity tracking in a question\-answering format, without deepening the underlying capability\.

### 4\.5Entity tracking is robust to semantic content

A distinguishing feature of our design is the use of naturalistic narratives rather than the procedural formats of prior work\(Kim and Schuster,[2023](https://arxiv.org/html/2608.18083#bib.bib18); Kimet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib20)\)\. While this makes the task more representative of natural comprehension, it also raises the possibility that models exploit lexical associations – e\.g\., inferring a key is in a box because keys and boxes co\-occur – rather than tracking entity movements\. To control for this, we evaluated all models on three template conditions: standard objects, pseudowords, and semantically improbable objects \([Figure 6](https://arxiv.org/html/2608.18083#S4.F6); panel A\)\.

Performance was consistent across conditions \([Figure 6](https://arxiv.org/html/2608.18083#S4.F6); panels B,C\)\. Accuracy remained within 3 percentage points of the standard template across all model families and evaluation formats\. As a manipulation check, we compared sequence log\-probabilities across templates and confirmed that models did register the lexical manipulation: pseudoword narratives were101710^\{17\}to102710^\{27\}times less likely than standard narratives, and improbable object narratives were10810^\{8\}to101610^\{16\}times less likely across all evaluated models\. The fact that accuracy was unaffected despite this increased surprisal indicates that entity tracking operates over discourse structure rather than lexical familiarity\.

## 5Discussion

We propose a new, controlled paradigm for evaluating entity tracking via naturalistic narratives\. Our findings challenge and refine key conclusions from prior work on entity tracking in language models\.

The most influential prior evaluation of entity tracking – the boxes paradigm\(Kim and Schuster,[2023](https://arxiv.org/html/2608.18083#bib.bib18); Kimet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib20)\)– requires tracking objects through sequences of movements along abstract locations \(Box 1, Box 2, etc\)\. For humans, solving this task requires step\-by\-step reasoning and would be difficult without extended time and external aids such as pen and paper\. This is fundamentally different from the implicit, on\-the\-fly updating that characterises entity tracking in natural language comprehension\. Our paradigm targets this latter process using naturalistic narratives and speeded responses\. Even under this more intuitive task, however, we observe that human entity tracking is far from perfect, underscoring the importance of measuring humans when evaluating cognitive abilities of language modelsIvanova \([2025](https://arxiv.org/html/2608.18083#bib.bib39)\)\.

Importantly, in this more naturalistic paradigm, we observe that language models perform far better than previously reported\. On the implicit task – which, especially for smaller models, provides a more sensitive measure of underlying capacities than explicit generation\(Hu and Frank,[2024](https://arxiv.org/html/2608.18083#bib.bib9)\)– human\-level entity tracking is already present at 410M parameters\. This stands in stark contrast to prior claimsKimet al\.\([2024](https://arxiv.org/html/2608.18083#bib.bib20)\); Kim and Schuster \([2023](https://arxiv.org/html/2608.18083#bib.bib18)\)that it requires at least 13B parameters with 500B additional code specialised tokens \(Code Llama\), or 7B parameters with 2T code tokens \(DeepSeek Coder\)\. Pythia 410M is over an order of magnitude smaller in parameter scale than the smallest code\-augmented model previously found sufficient, and trained on just 300B tokens of predominantly natural language\. This suggests that the scale and data requirements reported by prior work may reflect the procedural demands of their task rather than fundamental limits on entity tracking\.

The fact that previous studies found code pretraining to be crucial for entity tracking\(Kim and Schuster,[2023](https://arxiv.org/html/2608.18083#bib.bib18); Kimet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib20)\)aligns with a broader set of findings that code pretraining can improve many non\-code downstream abilities\(Madaanet al\.,[2022](https://arxiv.org/html/2608.18083#bib.bib5); Pettyet al\.,[2025](https://arxiv.org/html/2608.18083#bib.bib4); Aryabumiet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib3)\)\. However, code related improvements are typically seen in relatively formal domains such as reasoning\(Madaanet al\.,[2022](https://arxiv.org/html/2608.18083#bib.bib5); Aryabumiet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib3); Ruiset al\.,[2025](https://arxiv.org/html/2608.18083#bib.bib2)\)and semantic parsing tasks\(Pettyet al\.,[2025](https://arxiv.org/html/2608.18083#bib.bib4)\)that differ substantially from naturalistic language comprehension\. This raises the question whether code pretraining is critical for entity tracking itself, or specifically for the procedural, program\-like operationalisation used in prior studies\. Our results cannot conclusively answer this \(see Limitations\), but at minimum demonstrate that the requirements for entity tracking depend on how it is measured – and that for naturalistic texts, the bar appears lower than previously thought\.

## Limitations

There are three important limitations to the conclusions drawn in this work\. First, we only evaluate existing pretrained models\. Although these are fully open, both Pythia\(trained on the Pile; Bidermanet al\.,[2023](https://arxiv.org/html/2608.18083#bib.bib23)\)and OLMo 2\(Team OLMo,[2025](https://arxiv.org/html/2608.18083#bib.bib24)\)include code in their pretraining data, and we therefore cannot isolate the specific contribution of code to the entity tracking capacity we observe\. Controlled ablations comparing text\-only against code\-augmented pretraining, or targeted analyses tracing the influence of specific pretraining data\(e\.g\., Ruiset al\.,[2025](https://arxiv.org/html/2608.18083#bib.bib2)\), would be needed to determine whether naturalistic entity tracking emerges without any code exposure\.

Second, our output\-based evaluation cannot identify the underlying algorithms models use\. Future mechanistic interpretability work could test whether the entity tracking circuits and algorithms identified in structured tasks\(Prakashet al\.,[2024](https://arxiv.org/html/2608.18083#bib.bib19); Liet al\.,[2025](https://arxiv.org/html/2608.18083#bib.bib7)\)also operate during naturalistic language processing, or whether they use different strategies for discourse\-based entity tracking, or switch between algorithmic state\-tracking and shallower, ’computationally cheaper’ heuristics – analogous to the “end\-game degradation” observed in game\-playing modelsNandaet al\.\([2023](https://arxiv.org/html/2608.18083#bib.bib1)\)\.

Third, our evaluation prioritised comparability between humans and models: we tested complexity levels at which humans still perform above chance, and used smaller, fully open model families to trace scaling effects and the role of instruction tuning\. Contemporary models – Llama 3\.3\(Llama Team,[2024](https://arxiv.org/html/2608.18083#bib.bib35)\)and Qwen 2\.5\(Qwen Team,[2024](https://arxiv.org/html/2608.18083#bib.bib36)\)– were included only as high\-performing reference points and saturated the task at all tested levels\. Evaluating contemporary models to their breaking point at multiple model scales, and characterising the performance when models \(and humans\) are allowed to do more extended reasoning, remains an interesting direction for future work\.

## Acknowledgements

This work has been partly funded by the Dutch Research Council \(NWO\) under Veni grant VI\.Veni\.231G\.043 to MH\.

## References

- Allen Institute for AI \(2025\)OLMo 2 32B: First fully open model to outperform GPT 3\.5 and GPT 4o mini\.Note:Ai2 BlogExternal Links:[Link](https://allenai.org/blog/olmo2-32B)Cited by:[§4\.3](https://arxiv.org/html/2608.18083#S4.SS3.p3.1)\.
- V\. Aryabumi, Y\. Su, R\. Ma, A\. Morisot, I\. Zhang, A\. Locatelli, M\. Fadaee, A\. Üstün, and S\. Hooker \(2024\)To code, or not to code? exploring impact of code in pre\-training\.arXiv preprint arXiv:2408\.10914\.External Links:[Link](https://arxiv.org/abs/2408.10914)Cited by:[§5](https://arxiv.org/html/2608.18083#S5.p4.1)\.
- E\. M\. Bender and A\. Koller \(2020\)Climbing towards nlu: on meaning, form, and understanding in the age of data\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 5185–5198\.External Links:[Link](https://aclanthology.org/2020.acl-main.463/)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. van der Wal \(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2304.01373)Cited by:[§3\.4](https://arxiv.org/html/2608.18083#S3.SS4.p1.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p1.1)\.
- F\. Ferreira, K\. G\. Bailey, and V\. Ferraro \(2002\)Good\-enough representations in language comprehension\.Current directions in psychological science11\(1\),pp\. 11–15\.External Links:[Link](https://doi.org/10.1111/1467-8721.00158)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p3.1)\.
- J\. Groenendijk and M\. Stokhof \(1991\)Dynamic predicate logic\.Linguistics and Philosophy14\(1\),pp\. 39–100\.External Links:[Document](https://dx.doi.org/10.1007/BF00628304)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1)\.
- I\. Heim \(2002\)File change semantics and the familiarity theory of definiteness\.InFormal Semantics: The Essential Readings,P\. Portner and B\. H\. Partee \(Eds\.\),pp\. 223–248\.Note:Originally published in 1983Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1)\.
- J\. Hu and M\. C\. Frank \(2024\)Auxiliary task demands mask the capabilities of smaller language models\.InProceedings of the 1st Conference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=U5BUzSn4tD)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.18083#S3.SS1.p3.1),[§5](https://arxiv.org/html/2608.18083#S5.p3.1)\.
- A\. A\. Ivanova \(2025\)How to evaluate the cognitive abilities of LLMs\.Nature Human Behaviour9,pp\. 230–233\.External Links:[Document](https://dx.doi.org/10.1038/s41562-024-02096-z)Cited by:[§5](https://arxiv.org/html/2608.18083#S5.p2.1)\.
- E\. Keuleers and M\. Brysbaert \(2010\)Wuggy: a multilingual pseudoword generator\.Behavior Research Methods42\(3\),pp\. 627–633\.External Links:[Document](https://dx.doi.org/10.3758/BRM.42.3.627)Cited by:[§3\.2](https://arxiv.org/html/2608.18083#S3.SS2.p3.1)\.
- N\. Kim, S\. Schuster, and S\. Toshniwal \(2024\)Code pretraining improves entity tracking abilities of language models\.arXiV preprint arXiV:2405\.21068\.External Links:[Link](https://arxiv.org/abs/2405.21068)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p2.1),[§1](https://arxiv.org/html/2608.18083#S1.p3.1),[§2](https://arxiv.org/html/2608.18083#S2.p3.1),[§4\.5](https://arxiv.org/html/2608.18083#S4.SS5.p1.1),[§5](https://arxiv.org/html/2608.18083#S5.p2.1),[§5](https://arxiv.org/html/2608.18083#S5.p3.1),[§5](https://arxiv.org/html/2608.18083#S5.p4.1)\.
- N\. Kim and S\. Schuster \(2023\)Entity tracking in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2305.02363)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p2.1),[§1](https://arxiv.org/html/2608.18083#S1.p3.1),[§2](https://arxiv.org/html/2608.18083#S2.p3.1),[§4\.5](https://arxiv.org/html/2608.18083#S4.SS5.p1.1),[§5](https://arxiv.org/html/2608.18083#S5.p2.1),[§5](https://arxiv.org/html/2608.18083#S5.p3.1),[§5](https://arxiv.org/html/2608.18083#S5.p4.1)\.
- B\. Z\. Li, Z\. C\. Guo, and J\. Andreas \(2025\)\(How\) Do Language Models Track State?\.Forty\-second International Conference on Machine Learning\.External Links:[Link](https://icml.cc/virtual/2025/poster/46279)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p2.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p2.1)\.
- B\. Z\. Li, M\. Nye, and J\. Andreas \(2021\)Implicit representations of meaning in neural language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1813–1827\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.143),[Link](https://"https//aclanthology.org/2021.acl-long.143/%22)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1),[§1](https://arxiv.org/html/2608.18083#S1.p2.1)\.
- K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2022\)Emergent world representations: exploring a sequence model trained on a synthetic task\.arXiv preprint arXiv:2210\.13382\.External Links:[Link](https://arxiv.org/abs/2210.13382)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p1.1)\.
- Llama Team \(2024\)The Llama 3 Herd of Models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.4](https://arxiv.org/html/2608.18083#S3.SS4.p1.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p3.1)\.
- A\. Madaan, S\. a\. Zhou, U\. Alon, Y\. Yang, and G\. Neubig \(2022\)Language models of code are few\-shot commonsense learners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 1384–1403\.External Links:[Link](https://arxiv.org/abs/2210.07128)Cited by:[§5](https://arxiv.org/html/2608.18083#S5.p4.1)\.
- N\. Nanda, A\. Lee, and M\. Wattenberg \(2023\)Emergent linear representations in world models of self\-supervised sequence models\.InProceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,pp\. 16–30\.External Links:[Link](https://aclanthology.org/2023.blackboxnlp-1.2/),[Document](https://dx.doi.org/10.18653/v1/2023.blackboxnlp-1.2)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p1.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p2.1)\.
- J\. Petty, S\. van Steenkiste, and T\. Linzen \(2025\)How does code pretraining affect language model task performance?\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/pdf?id=2sghJ1yYOr)Cited by:[§5](https://arxiv.org/html/2608.18083#S5.p4.1)\.
- N\. Prakash, T\. R\. Shaham, T\. Haklay, Y\. Belinkov, and D\. Bau \(2024\)Fine\-tuning enhances existing mechanisms: a case study on entity tracking\.arXiv preprint arXiv:2402\.14811\.External Links:[Link](https://arxiv.org/abs/2402.14811)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p2.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p2.1)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.4](https://arxiv.org/html/2608.18083#S3.SS4.p1.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p3.1)\.
- L\. Ruis, M\. Mozes, J\. Bae, S\. R\. Kamalakara, D\. Gnaneshwar, A\. Locatelli, R\. Kirk, T\. Rocktäschel, E\. Grefenstette, and M\. Bartolo \(2025\)Procedural knowledge in pretraining drives reasoning in large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 29367–29429\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/482847908fd916b5b6b9e82525c773ad-Paper-Conference.pdf)Cited by:[§5](https://arxiv.org/html/2608.18083#S5.p4.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p1.1)\.
- M\. Shanahan \(2024\)Talking about large language models\.Communications of the ACM67\(2\),pp\. 68–79\.External Links:[Link](https://doi.org/10.1145/3624724)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1)\.
- Team OLMo \(2025\)2 OLMo 2 furious\.arXiv preprint arXiv:2501\.00656\.External Links:[Link](https://arxiv.org/abs/2501.00656)Cited by:[§3\.4](https://arxiv.org/html/2608.18083#S3.SS4.p1.1),[Limitations](https://arxiv.org/html/2608.18083#Sx1.p1.1)\.
- K\. Vafa, P\. G\. Chang, A\. Rambachan, and S\. Mullainathan \(2025\)What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2507.06952)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p1.1)\.
- K\. Vafa, J\. Y\. Chen, A\. Rambachan, J\. Kleinberg, and S\. Mullainathan \(2024\)Evaluating the world model implicit in a generative model\.Advances in Neural Information Processing Systems37,pp\. 26941–26975\.External Links:[Link](https://dl.acm.org/doi/10.5555/3737916.3738762)Cited by:[§2](https://arxiv.org/html/2608.18083#S2.p1.1)\.
- S\. Wang, G\. Durrett, and K\. Erk \(2018\)Modeling semantic plausibility by injecting world knowledge\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 303–308\.External Links:[Link](https://aclanthology.org/N18-2049/)Cited by:[§3\.2](https://arxiv.org/html/2608.18083#S3.SS2.p3.1)\.
- R\. A\. Zwaan and G\. A\. Radvansky \(1998\)Situation models in language comprehension and memory\.Psychological Bulletin123\(2\),pp\. 162–185\.External Links:[Document](https://dx.doi.org/10.1037/0033-2909.123.2.162)Cited by:[§1](https://arxiv.org/html/2608.18083#S1.p1.1)\.

## Appendix APerformance of OLMo 2 Instruct Models on the Explicit Task

![Refer to caption](https://arxiv.org/html/2608.18083v1/figures/figE.png)Figure 7:OLMo 2 Instruct models on the explicit \(generation\) task\. Instruction\-tuned models show substantial scaling effects, with accuracy improving from 55% \(1B\) to 94% \(32B\) overall\. The 32B model maintains high accuracy even at C5, while smaller instruction\-tuned models show steep performance declines with increasing complexity\.
## Appendix BObject Words by Template Condition

Table 1:Object words used in each template condition\. The conditions include: 1\) common & possible objects \(standard template;N=12N=12\), 2\) pseudowords derived from the standard objects \(N=12N=12\), 3a\) rare but possible objects \(N=4N=4\), 3b\) rare and physically impossible objects \(size violation;N=4N=4\), and 3c\) rare and logically impossible objects \(abstract violation;N=4N=4\)\.
## Appendix CEvaluation Prompt

For the explicit evaluation, we used the following instruction prompt across all models:

Follow these instructions:1\. Carefully read the story and question to identify the object’s final location\.2\. Your response must contain ONLY the single word for that final location \(e\.g\., if the final location is ’the red box’, respond ’box’; if it’s ’the bag’, respond ’bag’\)\.3\. DO NOT include any reasoning, explanations, introductory phrases, sentences, or punctuation\. Your entire response should be just the single location word\.Location:"""

## Appendix DExample Narratives

Explicit Task

Complexity 1, Focus Object 0

Dora began tidying up her room\. She picked up a book and put it into the basket\. She paused to check her phone\. Q: The book is in the …

expected\_answer: basket

Complexity 2, Focus Object 0

Lila began tidying up her room\. She picked up a bookmark and placed it into the bag\. She moved the bookmark from the bag to the drawer\. She glanced out the window\. Q: The bookmark is in the …

expected\_answer: drawer

Complexity 3, Focus Object 0

Sophia began tidying up her room\. She picked up a key and carefully placed it into the blue basket, where she noticed a photo\. Curious, she moved the key from the blue basket to the modern box\. Then, she moved the photo from the blue basket to the vintage bag\. She glanced out the window\. Q: The key is in the …

expected\_answer: box

Complexity 3, Focus Object 1

Lila began tidying up her room\. She picked up a postcard and carefully placed it into the vintage drawer, where she noticed a letter\. Curious, she moved the postcard from the vintage drawer to the box\. Then, she moved the letter from the vintage drawer to the bag\. She took a sip of water\. Q: The letter is in the …

expected\_answer: bag

Complexity 4, Focus Object 0

Anna began tidying up her room\. She picked up a postcard and carefully placed it into the box\. While organizing, she noticed a scarf in the basket\. Curious, she moved the scarf to the vintage drawer and found a watch there\. She put the watch into the box\. She paused to check her phone\. Q: The postcard is in the …

expected\_answer: box

Complexity 4, Focus Object 1

Mia began tidying up her room\. She picked up a pen and carefully placed it into the small box\. While organizing, she noticed a coin in the colorful drawer\. Curious, she moved the coin to the colorful basket and found a wallet there\. She put the wallet into the small box\. She paused for a moment to stretch\. Q: The coin is in the …

expected\_answer: basket

Complexity 4, Focus Object 2

Emma began tidying up her room\. She picked up a photo and carefully placed it into the basket\. While organizing, she noticed a pen in the box\. Curious, she moved the pen to the drawer and found a watch there\. She put the watch into the basket\. She paused for a moment to stretch\. Q: The watch is in the …

expected\_answer: basket

Complexity 5, Focus Object 0

Sophia began tidying up her room\. She picked up a pen and carefully placed it into the bag\. While organizing, she noticed a ring in the box\. Curious, she moved the ring to the drawer and found a photo there\. She picked up the photo and put it into the basket\. Then, Sophia moved the scarf from the basket to the box\. She paused for a moment to stretch\. Q: The pen is in the …

expected\_answer: bag

Complexity 5, Focus Object 1

Lila began tidying up her room\. She picked up a book and carefully placed it into the modern basket\. While organizing, she noticed a letter in the drawer\. Curious, she moved the letter to the bag and found a coin there\. She picked up the coin and put it into the box\. Then, Lila moved the photo from the box to the drawer\. She took a sip of water\. Q: The letter is in the …

expected\_answer: bag

Complexity 5, Focus Object 2

Anna began tidying up her room\. She picked up a bookmark and carefully placed it into the basket\. While organizing, she noticed a letter in the bag\. Curious, she moved the letter to the vintage drawer and found a key there\. She picked up the key and put it into the modern box\. Then, Anna moved the ring from the modern box to the bag\. She paused to check her phone\. Q: The key is in the …

expected\_answer: box

Complexity 5, Focus Object 3

Emma began tidying up her room\. She picked up a wallet and carefully placed it into the bag\. While organizing, she noticed a key in the vintage box\. Curious, she moved the key to the drawer and found a watch there\. She picked up the watch and put it into the basket\. Then, Emma moved the postcard from the basket to the vintage box\. She paused for a moment to stretch\. Q: The postcard is in the …

expected\_answer: box

Implicit Task

Complexity 1, Focus Object 0

prompt\_good: Dora began tidying up her room\. She picked up a photo and put it into the modern bag\. She glanced out the window\. Later, Dora picked up the photo from the bag and placed it into the drawer\.

prompt\_bad: Dora began tidying up her room\. She picked up a photo and put it into the modern bag\. She glanced out the window\. Later, Dora picked up the photo from the drawer and placed it into the box\.

Complexity 2, Focus Object 0

prompt\_good: Emma began tidying up her room\. She picked up a wallet and placed it into the bag\. She moved the wallet from the bag to the box\. She glanced out the window\. Later, Emma picked up the wallet from the box and placed it into the bag\.

prompt\_bad: Emma began tidying up her room\. She picked up a wallet and placed it into the bag\. She moved the wallet from the bag to the box\. She glanced out the window\. Later, Emma picked up the wallet from the bag and placed it into the basket\.

Complexity 3, Focus Object 0

prompt\_good: Dora began tidying up her room\. She picked up a watch and carefully placed it into the small bag, where she noticed a wallet\. Curious, she moved the watch from the small bag to the basket\. Then, she moved the wallet from the small bag to the box\. She paused to check her phone\. Later, Dora picked up the watch from the basket and placed it into the bag\.

prompt\_bad: Dora began tidying up her room\. She picked up a watch and carefully placed it into the small bag, where she noticed a wallet\. Curious, she moved the watch from the small bag to the basket\. Then, she moved the wallet from the small bag to the box\. She paused to check her phone\. Later, Dora picked up the watch from the bag and placed it into the box\.

Complexity 3, Focus Object 1

prompt\_good: Mia began tidying up her room\. She picked up a bookmark and carefully placed it into the blue drawer, where she noticed a key\. Curious, she moved the bookmark from the blue drawer to the basket\. Then, she moved the key from the blue drawer to the bag\. She paused to check her phone\. Later, Mia picked up the key from the bag and placed it into the box\.

prompt\_bad: Mia began tidying up her room\. She picked up a bookmark and carefully placed it into the blue drawer, where she noticed a key\. Curious, she moved the bookmark from the blue drawer to the basket\. Then, she moved the key from the blue drawer to the bag\. She paused to check her phone\. Later, Mia picked up the key from the box and placed it into the drawer\.

Complexity 4, Focus Object 0

prompt\_good: Sophia began tidying up her room\. She picked up a pen and carefully placed it into the box\. While organizing, she noticed a scarf in the drawer\. Curious, she moved the scarf to the bag and found a ring there\. She put the ring into the box\. She glanced out the window\. Later, Sophia picked up the pen from the box and placed it into the basket\.

prompt\_bad: Sophia began tidying up her room\. She picked up a pen and carefully placed it into the box\. While organizing, she noticed a scarf in the drawer\. Curious, she moved the scarf to the bag and found a ring there\. She put the ring into the box\. She glanced out the window\. Later, Sophia picked up the pen from the basket and placed it into the drawer\.

Complexity 4, Focus Object 1

prompt\_good: Anna began tidying up her room\. She picked up a bookmark and carefully placed it into the small bag\. While organizing, she noticed a watch in the basket\. Curious, she moved the watch to the drawer and found a letter there\. She put the letter into the small bag\. She paused for a moment to stretch\. Later, Anna picked up the watch from the drawer and placed it into the bag\.

prompt\_bad: Anna began tidying up her room\. She picked up a bookmark and carefully placed it into the small bag\. While organizing, she noticed a watch in the basket\. Curious, she moved the watch to the drawer and found a letter there\. She put the letter into the small bag\. She paused for a moment to stretch\. Later, Anna picked up the watch from the bag and placed it into the basket\.

Complexity 4, Focus Object 2

prompt\_good: Lila began tidying up her room\. She picked up a coin and carefully placed it into the small drawer\. While organizing, she noticed a book in the basket\. Curious, she moved the book to the bag and found a watch there\. She put the watch into the small drawer\. She paused to check her phone\. Later, Lila picked up the watch from the drawer and placed it into the basket\.

prompt\_bad: Lila began tidying up her room\. She picked up a coin and carefully placed it into the small drawer\. While organizing, she noticed a book in the basket\. Curious, she moved the book to the bag and found a watch there\. She put the watch into the small drawer\. She paused to check her phone\. Later, Lila picked up the watch from the basket and placed it into the bag\.

Complexity 5, Focus Object 0

prompt\_good: Lila began tidying up her room\. She picked up a scarf and carefully placed it into the blue box\. While organizing, she noticed a letter in the red drawer\. Curious, she moved the letter to the bag and found a postcard there\. She picked up the postcard and put it into the vintage basket\. Then, Lila moved the photo from the vintage basket to the red drawer\. She paused to check her phone\. Later, Lila picked up the scarf from the box and placed it into the basket\.

prompt\_bad: Lila began tidying up her room\. She picked up a scarf and carefully placed it into the blue box\. While organizing, she noticed a letter in the red drawer\. Curious, she moved the letter to the bag and found a postcard there\. She picked up the postcard and put it into the vintage basket\. Then, Lila moved the photo from the vintage basket to the red drawer\. She paused to check her phone\. Later, Lila picked up the scarf from the basket and placed it into the drawer\.

Complexity 5, Focus Object 1

prompt\_good: Lila began tidying up her room\. She picked up a watch and carefully placed it into the blue basket\. While organizing, she noticed a wallet in the box\. Curious, she moved the wallet to the drawer and found a key there\. She picked up the key and put it into the bag\. Then, Lila moved the coin from the bag to the box\. She paused to check her phone\. Later, Lila picked up the wallet from the drawer and placed it into the bag\.

prompt\_bad: Lila began tidying up her room\. She picked up a watch and carefully placed it into the blue basket\. While organizing, she noticed a wallet in the box\. Curious, she moved the wallet to the drawer and found a key there\. She picked up the key and put it into the bag\. Then, Lila moved the coin from the bag to the box\. She paused to check her phone\. Later, Lila picked up the wallet from the bag and placed it into the box\.

Complexity 5, Focus Object 2

prompt\_good: Mia began tidying up her room\. She picked up a coin and carefully placed it into the basket\. While organizing, she noticed a watch in the modern box\. Curious, she moved the watch to the drawer and found a wallet there\. She picked up the wallet and put it into the bag\. Then, Mia moved the ring from the bag to the modern box\. She paused for a moment to stretch\. Later, Mia picked up the wallet from the bag and placed it into the basket\.

prompt\_bad: Mia began tidying up her room\. She picked up a coin and carefully placed it into the basket\. While organizing, she noticed a watch in the modern box\. Curious, she moved the watch to the drawer and found a wallet there\. She picked up the wallet and put it into the bag\. Then, Mia moved the ring from the bag to the modern box\. She paused for a moment to stretch\. Later, Mia picked up the wallet from the basket and placed it into the box\.

Complexity 5, Focus Object 3

prompt\_good: Dora began tidying up her room\. She picked up a scarf and carefully placed it into the bag\. While organizing, she noticed a pen in the drawer\. Curious, she moved the pen to the big box and found a ring there\. She picked up the ring and put it into the small basket\. Then, Dora moved the watch from the small basket to the drawer\. She took a sip of water\. Later, Dora picked up the watch from the drawer and placed it into the box\.

prompt\_bad: Dora began tidying up her room\. She picked up a scarf and carefully placed it into the bag\. While organizing, she noticed a pen in the drawer\. Curious, she moved the pen to the big box and found a ring there\. She picked up the ring and put it into the small basket\. Then, Dora moved the watch from the small basket to the drawer\. She took a sip of water\. Later, Dora picked up the watch from the box and placed it into the basket\.