Thought without systematicity? Evaluating reasoning models on rule induction tasks

# Thought without systematicity?Evaluating reasoning models on rule induction tasks
Source: [https://arxiv.org/html/2609.13948](https://arxiv.org/html/2609.13948)
###### Abstract

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept\. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task\. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models\. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution\. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task\. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in\.

Figure 1:Systematicity of reasoning models on rule induction tasks\.We evaluate reasoning models on compositional families of rule induction tasks\. For each particular task within a family, we generateK=5K=5structurally equivalent task variants per type of task invariance\. For each set of task variations, we report whether any of the variations was solved correctly, the fraction of variations solved correctly and whether all variations were solved correctly and report the average over task variation sets within a task family\. We find that almost all models are perfectly systematic on the Boolean category learning task family, but see a notable systematicity gap between the ability to solve any task variation and solving all task variations on the remaining task families\.## 1Introduction

Figure 2:Evaluating systematicity in rule induction tasks with compositional structure\.\(A\) Compositional task families allow to construct large numbers of tasks with related structure\. The shown task is a simplified version of the grammar\-based instruction\-learning task presented in Section[4\.1](https://arxiv.org/html/2609.13948#S4.SS1)\. \(B\) Each rule induction task requires inferring hidden rules on a set of input→\\rightarrowoutput support examples to solve novel query examples\. \(C\) The compositional task structure allows creating systematic variations of structurally equivalent tasks\. \(D\) Reasoning models can be independently evaluated across structurally equivalent task variants to assess their systematicity\.Systematicity is a defining feature of human language and thought: When we can understand one concept it implies that we can also understand many closely related concepts\([Fodor and Pylyshyn,, 1988](https://arxiv.org/html/2609.13948#bib.bib17);[McLaughlin,, 1993](https://arxiv.org/html/2609.13948#bib.bib34)\)\. For instance, when we learn how toblicket, we can alsoblickettwice, and if we are able to learn that all red squares arewudsywe are equally able to learn that all blue circles arewudsy\. This systematicity reflects humans’ flexibility to effortlessly generate and comprehend unseen variations of familiar elements\([Chomsky,, 1985](https://arxiv.org/html/2609.13948#bib.bib12);[Lake et al\.,, 2017](https://arxiv.org/html/2609.13948#bib.bib30)\)and it is a crucial assumption underlying cognitive tests, because it enables measuring an ability in a particular context but make inferences about that ability beyond the specific context within which it was measured\.

The systematicity of human cognition has many facets reflecting the various ways in which a task can be altered according to its underlying compositional structure\([Hupkes et al\.,, 2020](https://arxiv.org/html/2609.13948#bib.bib22)\)\. This includes substitutions such as using a different symbol for the same underlying concept, or recombinations of the parts of an expression that serve similar roles\. These invariances reflect compositional task structure relating tasks to each other through a common syntax that prescribes how sets of primitives can be composed into different configurations\. From this perspective, systematicity can be understood as a behavioral outcome of successfully capturing the underlying compositional structure of a family of tasks\. Compositional task families are exponential in nature and it is virtually impossible to exhaustively include all possible task variations in the training data even at large scale\([Schug,, 2025](https://arxiv.org/html/2609.13948#bib.bib44)\)\. This raises the question:Do reasoning models exhibit systematicity of thought and perform consistently across the many possible, structurally equivalent variations of a given task?

Here, we study this question by drawing on established experimental paradigms from cognitive science\. In particular, we consider rule induction tasks that require the learner to infer latent structure, and evaluate it across structurally equivalent task variations to probe the systematicity of its thinking as illustrated in Figure[2](https://arxiv.org/html/2609.13948#S1.F2)\. A variety of such tasks have been studied in cognitive science, ranging from simple Boolean concept learning, abstract reasoning over symbolic sequences, to discovering the syntax and semantics of artificial languages\([Raven,, 1962](https://arxiv.org/html/2609.13948#bib.bib40);[Feldman,, 2000](https://arxiv.org/html/2609.13948#bib.bib15);[Piantadosi et al\.,, 2016](https://arxiv.org/html/2609.13948#bib.bib39);[Lake and Baroni,, 2023](https://arxiv.org/html/2609.13948#bib.bib29);[Rule et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib43)\)\. These tasks have compositional structure that allows us to generate a large number of possible tasks and probe understanding beyond the specifics of any particular experience\. Importantly, for each specific task we can create structurally equivalent task variations by usingtask isomorphisms– equivariant task transformations based on the underlying compositional structure – to evaluate systematicity\. Unlike humans whose mental states are invariably altered by experience, models provide the opportunity to evaluate systematicity in its strictest form by observing behavioral outcomes in multiple counterfactual worlds that do not affect each other\.

We summarize ourmain contributionsin the following:•In Section[3](https://arxiv.org/html/2609.13948#S3), we formalize systematicity and how to measure it over structurally equivalent rule induction tasks via task isomorphisms\.•In Section[4](https://arxiv.org/html/2609.13948#S4), we adapt four rule induction task families for generating systematic task variations\.–Section[4\.1](https://arxiv.org/html/2609.13948#S4.SS1): Grammar\-based instruction learning\([Lake and Baroni,, 2023](https://arxiv.org/html/2609.13948#bib.bib29)\)–Section[4\.2](https://arxiv.org/html/2609.13948#S4.SS2): A symbolic version of Raven’s progressive matrices\([Schug et al\.,, 2025](https://arxiv.org/html/2609.13948#bib.bib45)\)–Section[4\.3](https://arxiv.org/html/2609.13948#S4.SS3): Program induction over integer sequences\([Rule et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib43)\)–Section[4\.4](https://arxiv.org/html/2609.13948#S4.SS4): Boolean category learning\([Piantadosi et al\.,, 2016](https://arxiv.org/html/2609.13948#bib.bib39)\)•In Section[5](https://arxiv.org/html/2609.13948#S5), we evaluate the systematicity of reasoning models across task variations on our four task families and investigate the impact of reasoning effort and sampling noise\.

## 2Related work

The observation that human cognition is systematic has been a longstanding locus of inquiry, reaching back to early attempts to formulate laws of thought\([Boole,, 1854](https://arxiv.org/html/2609.13948#bib.bib4);[Griffiths,, 2026](https://arxiv.org/html/2609.13948#bib.bib19)\)\. Systematicity is a central tenet of the language of thought hypothesis – a prominent theory of human cognition which assumes that mental representations have combinatorial syntax and semantics\([Fodor,, 1975](https://arxiv.org/html/2609.13948#bib.bib16);[Fodor and Pylyshyn,, 1988](https://arxiv.org/html/2609.13948#bib.bib17);[Penn et al\.,, 2008](https://arxiv.org/html/2609.13948#bib.bib38);[Goodman et al\.,, 2015](https://arxiv.org/html/2609.13948#bib.bib18)\)\. Cognitive science therefore has a rich history of developing tests of rule learning, with varying latent syntactic and semantic structure, to investigate the language of thought hypothesis\. Classic work by[Feldman, \(2000\)](https://arxiv.org/html/2609.13948#bib.bib15)uses Boolean concept learning tasks to show that subjective difficulty is directly related to the complexity of the underlying rule and learning tasks with more complex logical rules have been used to investigate what concrete primitives might underlie the language of thought\([Piantadosi et al\.,, 2016](https://arxiv.org/html/2609.13948#bib.bib39);[Rule et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib43)\)\. Tasks that require inferring compositional string rewriting rules have further been used to compare compositional learning between humans and neural networks, a challenge neural networks have historically struggled with but recently made notable progress on\([Lake and Baroni,, 2023](https://arxiv.org/html/2609.13948#bib.bib29)\)\.

Despite this progress, early large language models were brittle and could be derailed by subtle variations in their inputs, including typos, irrelevant sentences, changes in phrasing, and reordering of examples\([Jia and Liang,, 2017](https://arxiv.org/html/2609.13948#bib.bib26);[Jiang et al\.,, 2020](https://arxiv.org/html/2609.13948#bib.bib27);[Zhao et al\.,, 2021](https://arxiv.org/html/2609.13948#bib.bib53);[Lu et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib32)\)\. While instruction tuning via human feedback\([Ouyang et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib37)\), progressive scaling\([Kaplan et al\.,, 2020](https://arxiv.org/html/2609.13948#bib.bib28);[Bubeck et al\.,, 2023](https://arxiv.org/html/2609.13948#bib.bib6)\)and inference\-time reasoning\([Wei et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib52);[Wang et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib49);[Lightman et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib31);[OpenAI et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib36)\)have rendered the resulting models significantly less sensitive to surface changes, adversarial brittleness remains a central concern\([Wang et al\.,, 2023](https://arxiv.org/html/2609.13948#bib.bib48);[Zou et al\.,, 2023](https://arxiv.org/html/2609.13948#bib.bib54);[Berglund et al\.,, 2024](https://arxiv.org/html/2609.13948#bib.bib2);[Romanou et al\.,, 2026](https://arxiv.org/html/2609.13948#bib.bib42);[Mondorf et al\.,, 2026](https://arxiv.org/html/2609.13948#bib.bib35);[Burnell et al\.,, 2026](https://arxiv.org/html/2609.13948#bib.bib7)\), in particular in the context of safety and alignment\([Wei et al\.,, 2023](https://arxiv.org/html/2609.13948#bib.bib51);[Chen et al\.,, 2025](https://arxiv.org/html/2609.13948#bib.bib9)\)\.

In recent years, stronger emphasis has therefore been put on evaluating model consistency through metamorphic testing where a known relationship between outputs of related inputs is evaluated\([Segura et al\.,, 2016](https://arxiv.org/html/2609.13948#bib.bib46)\)\. This includes consistency across languages\([Cho et al\.,, 2025](https://arxiv.org/html/2609.13948#bib.bib10)\)or category\-based substitution\([Ribeiro et al\.,, 2020](https://arxiv.org/html/2609.13948#bib.bib41)\)\. Logical consistency testing in particular evaluates the internal consistency of model knowledge\([Jang et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib24)\), e\.g\. by verifying invariance to paraphrasing\([Elazar et al\.,, 2021](https://arxiv.org/html/2609.13948#bib.bib13)\)or whether predicted relations are closed under transitivity\([Jang and Lukasiewicz,, 2023](https://arxiv.org/html/2609.13948#bib.bib25)\)\. Our systematicity evaluation can be interpreted as a specific type of metamorphic test that verifies invariant task\-solving ability with respect to a compositional data generating procedure\.

## 3What is systematicity and how can we measure it?

In the following, we formalize systematicity, how it can be measured in the setting of rule induction tasks and develop metrics to quantify a learner’s systematicity on such tasks\.

### 3\.1What is systematicity?

We define systematicity as invariance of ability with respect to certain transformations of a task with compositional structure\. For example, we might expect the ability to understand an expression like "Alice loves John" to be invariant to a permutation of its constituents like "John loves Alice"\. This means we treat systematicity as a latent property of behavior that makes assumptions about the compositional structure underlying a data generating process\. To quantify systematicity, we will measure how the ability to solve a task varies across task variations that modify the constituents of a task according to its compositional structure\. Since some task variations might change the difficulty of a task and confound the systematicity measure, we will restrict task variations to be structurally equivalent to each other through isomorphisms, one\-to\-one mappings between each task variation\.

### 3\.2Types of systematic task variations

The particular ways in which task constituents can be modified depend on the compositional structure of the data generating process\. In our task families we will consider the following task transformations:

- •Symbol substitution: Replacing symbols that carry no intrinsic semantic meaning relevant to the task\. For example replacing one pseudoword "dax" with another pseudoword "lug"\.
- •Feature rebinding: Rebinding latent variables to different features\. For example, representing the same number through size, orientation or numerosity\.
- •Constituent permutation: Shuffling constituents of the same type within an expression\. For example, "Alice loves John"↔\\leftrightarrow"John loves Alice"\.
- •Example reordering: Changing the order with which multiple independent examples are presented\.

### 3\.3Structurally equivalent rule induction tasks

We will evaluate systematicity within the setting of rule induction tasks as illustrated in Figure[2](https://arxiv.org/html/2609.13948#S1.F2)\. In each task, we present a learner with a set of input\-output examples based on which it has to infer hidden underlying rules\. We then verify whether the learner correctly inferred the rules by asking it to predict the outputs on a novel set of inputs\. For instance, in the Boolean category learning task, a learner might be presented with three support examples,→wudsy\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\}\{\}\{\}\{\}\{\{\}\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-4\.77 \-4\.77 M \-4\.77 \-4\.77 L \-4\.77 4\.77 L 4\.77 4\.77 L 4\.77 \-4\.77 Z M 4\.77 4\.77\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{wudsy\},→not wudsy\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\}\{\}\{\}\{\}\{\{\}\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-4\.77 \-4\.77 M \-4\.77 \-4\.77 L \-4\.77 4\.77 L 4\.77 4\.77 L 4\.77 \-4\.77 Z M 4\.77 4\.77\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{not wudsy\},→wudsy\\hbox to8\.15pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 4\.07498pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-5\.36 \-4\.77 L 5\.36 \-4\.77 L 0 4\.77 Z\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{wudsy\}, from which it has to infer the simplest rule that explains which objects arewudsy\(all red objects are wudsy\)\. We then evaluate whether it did so correctly by asking it to complete query examples such as,→?\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\}\{\}\{\}\{\}\{\{\}\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.3867,0\.6523,0\.6289\}\\lxSVG@begingroup@\{fill=\#63A6A0\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-4\.77 \-4\.77 M \-4\.77 \-4\.77 L \-4\.77 4\.77 L 4\.77 4\.77 L 4\.77 \-4\.77 Z M 4\.77 4\.77\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\textbf\{?\}and→?\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\textbf\{?\}\.

To study a learner’s systematicity across variations of a task of equal difficulty, we construct structurally equivalent tasks throughisomorphisms\. An isomorphism is a structure\-preserving mapping that is invertible and can therefore only alter the surface characteristics of a task but not its underlying structure\. For example, we can construct the structurally equivalent task→wudsy\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{wudsy\},→not wudsy\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.3867,0\.6523,0\.6289\}\\lxSVG@begingroup@\{fill=\#63A6A0\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{not wudsy\},→wudsy\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\}\{\}\{\}\{\}\{\{\}\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-4\.77 \-4\.77 M \-4\.77 \-4\.77 L \-4\.77 4\.77 L 4\.77 4\.77 L 4\.77 \-4\.77 Z M 4\.77 4\.77\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\texttt\{wudsy\},→?\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\textbf\{?\}and→?\\hbox to8\.15pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 4\.07498pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\\lx@inpgf@ignorespaces\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M \-5\.36 \-4\.77 L 5\.36 \-4\.77 L 0 4\.77 Z\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\rightarrow\\textbf\{?\}, where the hidden rule isall blue objects are wudsyand the colors and shapes were remapped accordingly\.

### 3\.4Isomorphic task variations

𝒳\{\\lx@inpgf@ignorespaces\\mathcal\{X\}\}𝒴\{\\lx@inpgf@ignorespaces\\mathcal\{Y\}\}𝒳~\{\\lx@inpgf@ignorespaces\\tilde\{\\mathcal\{X\}\}\}𝒴~\{\\lx@inpgf@ignorespaces\\tilde\{\\mathcal\{Y\}\}\}f\\scriptstyle\{\\lx@inpgf@ignorespaces f\}ϕ\\scriptstyle\{\\lx@inpgf@ignorespaces\\phi\}ψ\\scriptstyle\{\\lx@inpgf@ignorespaces\\psi\}f~\\scriptstyle\{\\lx@inpgf@ignorespaces\\tilde\{f\}\}Figure 3:Relation\-ship between the original and transformed hidden rule\.Formally, let𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}be input and output spaces\. We define a task as a tupleT=\(f,S,Q\)T=\(f,S,Q\), wheref:𝒳→𝒴f:\\mathcal\{X\}\\to\\mathcal\{Y\}is ahidden rule,S=\(\(xi,f⁡\(xi\)\)\)i=1NS=\\left\(\(x\_\{i\},f\(x\_\{i\}\)\)\\right\)\_\{i=1\}^\{N\}is thesupport set, andQ=\(xj′\)j=1MQ=\(x^\{\\prime\}\_\{j\}\)\_\{j=1\}^\{M\}is thequery set, withxi,xj′∈𝒳x\_\{i\},x^\{\\prime\}\_\{j\}\\in\\mathcal\{X\}\. Upon observing only the support setSS, a learner is tasked to predict the targets of the query set,f⁡\(xj′\)f\(x^\{\\prime\}\_\{j\}\)for allj=1,…,Mj=1,\\dots,M\. We say a learner solves a task if it correctly predicts the whole query set\.

Anisomorphic task variationis generated by applying a transformationτ=\(π,σ,ϕ,ψ\)\\tau=\(\\pi,\\sigma,\\phi,\\psi\), whereπ\\piandσ\\sigmaare permutations of the indices\{1,…,N\}\\\{1,\\dots,N\\\}and\{1,…,M\}\\\{1,\\dots,M\\\}respectively, andϕ:𝒳→𝒳~\\phi:\\mathcal\{X\}\\to\\tilde\{\\mathcal\{X\}\}andψ:𝒴→𝒴~\\psi:\\mathcal\{Y\}\\to\\tilde\{\\mathcal\{Y\}\}are bijections that map elements from the original input and output space to new input and output spaces𝒳~\\tilde\{\\mathcal\{X\}\}and𝒴~\\tilde\{\\mathcal\{Y\}\}\. Applyingτ\\tauyields a transformed taskτ⁡\(T\)=\(f~,S~,Q~\)\\tau\(T\)=\(\\tilde\{f\},\\tilde\{S\},\\tilde\{Q\}\), where the new hidden rule evaluates asf~=ψ∘f∘ϕ−1\\tilde\{f\}=\\psi\\circ f\\circ\\phi^\{\-1\}, the transformed and permuted support set isS~=\(ϕ⁡\(xπ⁡\(i\)\),ψ⁡\(f⁡\(xπ⁡\(i\)\)\)\)i=1N\\tilde\{S\}=\\left\(\\phi\(x\_\{\\pi\(i\)\}\),\\psi\(f\(x\_\{\\pi\(i\)\}\)\)\\right\)\_\{i=1\}^\{N\}, and the transformed and permuted query set isQ~=\(ϕ⁡\(xσ⁡\(j\)′\)\)j=1M\\tilde\{Q\}=\\left\(\\phi\(x^\{\\prime\}\_\{\\sigma\(j\)\}\)\\right\)\_\{j=1\}^\{M\}\.

### 3\.5Systematicity metrics

Figure 4:We use bullet charts to compactly visualize the three systematicity metrics\.A strictly systematic learner should be invariant to isomorphic task variations assuming that the semantics of any two input spaces are equally \(un\)informative for solving the task: If it can grasp the underlying structure well enough to solve one task variation, we would expect it to be able to solve them all\. Letpass⁡\(T\)∈\{0,1\}\\operatorname\{pass\}\(T\)\\in\\\{0,1\\\}be a binary indicator denoting whether a learner correctly predicts the targets for the entire query set of taskTT\. To evaluate a learner’s robustness to isomorphic variations, we sampleKKtask transformations\{τ1,…,τK\},τk∼p⁡\(τ\)\\\{\\tau\_\{1\},\\dots,\\tau\_\{K\}\\\},\\tau\_\{k\}\\sim p\(\\tau\)for each taskTTand calculate the following systematicity metrics:

- •Solve any ofKKvariations:maxk∈\{1,…,K\}⁡pass⁡\(τk​\(T\)\)\\max\_\{k\\in\\\{1,\\dots,K\\\}\}\\operatorname\{pass\}\(\\tau\_\{k\}\(T\)\)
- •Fraction of variations solved:1K​∑k=1Kpass⁡\(τk​\(T\)\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\operatorname\{pass\}\(\\tau\_\{k\}\(T\)\)
- •Solve allKKvariations:mink∈\{1,…,K\}⁡pass⁡\(τk​\(T\)\)\\min\_\{k\\in\\\{1,\\dots,K\\\}\}\\operatorname\{pass\}\(\\tau\_\{k\}\(T\)\)

Collectively, these three metrics allow us to characterize the systematicity with which a learner solves a task: If it never passes any task variation, this means the tasks are too difficult for the learner\. If it passes some task variations but not all of them, it is not fully systematic and the fraction of variations passed captures to what extent\. If all task variations are passed it can be considered systematic on this task with respect to the tested task variations\. We can take advantage of the fact that the three metrics are monotonically decreasing and visualize them as a bullet chart as shown in Figure[4](https://arxiv.org/html/2609.13948#S3.F4)\.

### 3\.6Systematicity under stochasticity

An important consideration in the context of systematicity is how to handle possible stochasticity in the learner that solves the tasks\. When repeatedly evaluating a stochastic learner on a given task, the learner might randomly fail to complete the task on some attempts\. While large language models were traditionally evaluated using greedy decoding, current reasoning models have been found to perform better when evaluated with stochastic sampling\([Wang et al\.,, 2022](https://arxiv.org/html/2609.13948#bib.bib49);[Guo et al\.,, 2025](https://arxiv.org/html/2609.13948#bib.bib20)\)\. In fact, many proprietary model providers now only allow to use reasoning models with a positive sampling temperature\. To account for the resulting decoding variance, we give each reasoning model multiple attempts per task variation \(here we useN=5N=5attempts throughout\) and consider a variation as passed if it was correctly solved in the majority of the attempts\. We will further study the impact of stochasticity on systematicity in Section[5\.3](https://arxiv.org/html/2609.13948#S5.SS3)\.

## 4Rule induction tasks for evaluating systematicity

In the following sections, we present four families of rule induction tasks built on established task paradigms from cognitive science\. Each data generating procedure relies on some form of compositional structure that allows to sample a large number of compositional tasks of varying difficulty\. Their synthetic nature provides us with the necessary control to create task variations that are guaranteed to be isomorphic in order to evaluate systematicity\.

### 4\.1Grammar\-based instruction\-learning

The grammar\-based instruction\-learning task family was introduced by[Lake and Baroni, \(2023\)](https://arxiv.org/html/2609.13948#bib.bib29)\. In each task from this family, the model must infer the latent grammatical rules of an artificial language from a few demonstrations to translate pseudolanguage commands into a sequence of outputs\. Since the artificial languages are procedurally generated from a meta\-grammar, an infinite number of such languages of varying complexity can in principle be generated\. This task family was originally designed to evaluate the ability of humans and neural networks for compositional generalization, the ability to solve unseen task compositions made from familiar parts\.

#### Instructions

To reduce the influence of prior experience on task performance as well as limit possible ambiguity, we provide detailed instructions on the general structure of each task in the system prompt, shown in Figure[11](https://arxiv.org/html/2609.13948#A3.F11)\.

#### Example

The following is a sample episode from the grammar\-based instruction\-learning task family\.

dax fep→​​\\displaystyle\\rightarrow\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}lug fep→​​\\displaystyle\\rightarrow\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}dax zup lug→​\\displaystyle\\rightarrow\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}dax zup lug fep→​​​\\displaystyle\\rightarrow\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}dax fep zup lug→​​​\\displaystyle\\rightarrow\\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.0313,0\.3164,0\.6133\}\\lxSVG@begingroup@\{fill=\#08519C\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\ \\hbox to7\.29pt\{\\vbox to7\.29pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 3\.64442pt\\lower\-3\.64442pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{\}\{\{\}\}\{\}\{\{\{\}\}\{\\lx@inpgf@ignorespaces\}\{\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\color\[rgb\]\{0\.9492,0\.3633,0\.3906\}\\lxSVG@begingroup@\{fill=\#F25D64\} \{\}\\lxSVG@fillstroke\\lxSVG@drawpath@unclipped\{M 0 0 M 4\.77 0 C 4\.77 2\.63 2\.63 4\.77 0 4\.77 C \-2\.63 4\.77 \-4\.77 2\.63 \-4\.77 0 C \-4\.77 \-2\.63 \-2\.63 \-4\.77 0 \-4\.77 C 2\.63 \-4\.77 4\.77 \-2\.63 4\.77 0 Z M 0 0\}\{\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\lug fep zup dax fep→?\\displaystyle\\rightarrow\\textbf\{?\}
To predict the single query example of this task, the learner must infer primitive rules, mapping input tokens, likedax, lug, to output tokens,RED, BLUE, and function rules that transform their inputs, likefep, zup\. Herefeptriples its input andzupswaps its inputs\. The learner then has to apply the inferred rules to a novel input composition and predict its answer, here\.

#### Task structure

Each generated language is governed by a uniquely sampled syntax that consists of primitive rules and function rules\. Whereas primitive rules are simple one\-to\-one mappings between input tokens \(e\.g\., pseudowords likedax,lug\) and output tokens \(e\.g\., capitalized colors likeRED,BLUE\), function rules specify how function tokens transform one or two adjacent arguments into a sequence of outputs\. The function arguments are either a single primitive token or match a whole preceding or succeeding string\. When applied, function rules deterministically reorder, delete or duplicate their arguments to produce a new output sequence\. To resolve syntactic ambiguities, functions that accept string arguments have a strict precedence order\. In addition, the parser evaluates sequences via left\-to\-right reduction, ensuring that for any valid input string there exists a unique translation\.

#### Support and query set

In the original grammar\-based instruction\-learning task family as used in[Lake and Baroni, \(2023\)](https://arxiv.org/html/2609.13948#bib.bib29), tasks were hand selected to ensure that their underlying grammar could be unambiguously inferred\. Since we would like to generate a large number of unambiguous tasks, we define a procedure to automatically create an instructive support set that ensures all primitive rules and function rules can be identified for a given, randomly sampled grammar \. Specifically, we construct the support set by producing lexical anchors that demonstrate the primitive rules \(e\.g\.,dax fep→\\rightarrowandlug fep→\\rightarrowin the example above\), template resolvers that reveal the arity and output transformations of the rule functions \(e\.g\.,dax zup lug→\\rightarrowshows thatzupis a two argument reversal function\), and precedence proofs that disambiguate the precedence order of multiple string matching function rules\. The query set as well as a configurable number of additional support examples is then generated from nested compositions that involve at least two function rule applications per example, ensuring that there are no duplicate examples across the support and query set\.

#### Task invariances

The resulting tasks have several invariances that we can use to generate isomorphic task variations\. We can permute the order of both the support and query examples \(example reordering\), apply a bijective mapping to the input and output vocabularies \(symbol substitution\) and permute the primitive tokens in a given expression, e\.g\. by changingdax zup lugtolug zup dax\(recomposition\)\.

### 4\.2Symbolic Raven’s progressive matrices

Raven’s progressive matrices is a classic human intelligence test\([Raven,, 1962](https://arxiv.org/html/2609.13948#bib.bib40)\)\. In its original form, each task consists of a three by three grid of abstract symbols with a missing final panel whose contents need to be inferred\.[Schug et al\., \(2025\)](https://arxiv.org/html/2609.13948#bib.bib45)introduce a symbolic variant of this task family that allows to procedurally generate Raven\-like tasks of varying difficulty, creating challenging abstract reasoning problems\.

#### Instructions

Similar to before we provide detailed instructions on the general structure of tasks from this task family in the system prompt, shown in Figure[12](https://arxiv.org/html/2609.13948#A3.F12)\.

#### Example

The following is a sample episode from the symbolic Raven’s progressive matrices task family using two \(F=2F=2\) and modulo 10 arithmetic\. The left side shows the unpermuted base task, while the right side additionally considers a column\-specific feature permutation which models the difficulty offinding correspondences\([Carpenter et al\.,, 1990](https://arxiv.org/html/2609.13948#bib.bib8)\)\.

\[52\]\[54\]\[56\]\[87\]\[89\]\[81\]\[35\]\[37\]\[??\]⇔\[25\]\[54\]\[65\]\[78\]\[89\]\[18\]\[53\]\[37\]\[??\]\\displaystyle\\begin\{array\}\[\]\{ccc\}\\begin\{bmatrix\}5\\\\ 2\\end\{bmatrix\}&\\begin\{bmatrix\}5\\\\ 4\\end\{bmatrix\}&\\begin\{bmatrix\}5\\\\ 6\\end\{bmatrix\}\\\\\[20\.00003pt\] \\begin\{bmatrix\}8\\\\ 7\\end\{bmatrix\}&\\begin\{bmatrix\}8\\\\ 9\\end\{bmatrix\}&\\begin\{bmatrix\}8\\\\ 1\\end\{bmatrix\}\\\\\[20\.00003pt\] \\begin\{bmatrix\}3\\\\ 5\\end\{bmatrix\}&\\begin\{bmatrix\}3\\\\ 7\\end\{bmatrix\}&\\begin\{bmatrix\}\\textbf\{?\}\\\\ \\textbf\{?\}\\end\{bmatrix\}\\end\{array\}\\quad\\Leftrightarrow\\quad\\begin\{array\}\[\]\{ccc\}\\begin\{bmatrix\}2\\\\ 5\\end\{bmatrix\}&\\begin\{bmatrix\}5\\\\ 4\\end\{bmatrix\}&\\begin\{bmatrix\}6\\\\ 5\\end\{bmatrix\}\\\\\[20\.00003pt\] \\begin\{bmatrix\}7\\\\ 8\\end\{bmatrix\}&\\begin\{bmatrix\}8\\\\ 9\\end\{bmatrix\}&\\begin\{bmatrix\}1\\\\ 8\\end\{bmatrix\}\\\\\[20\.00003pt\] \\begin\{bmatrix\}5\\\\ 3\\end\{bmatrix\}&\\begin\{bmatrix\}3\\\\ 7\\end\{bmatrix\}&\\begin\{bmatrix\}\\textbf\{?\}\\\\ \\textbf\{?\}\\end\{bmatrix\}\\end\{array\}To solve the unpermuted task \(left\), the learner must infer the rules applied horizontally to each aligned feature dimension\. Here, the first feature is constant \(8→\\rightarrow8→\\rightarrow8\), and the second feature follows an arithmetic progression of \+2 modulo 10 \(e\.g\.,7→\\rightarrow9→\\rightarrow1\)\. Applying these rules to the third row we obtain the target panel,\[39\]\\begin\{bmatrix\}3\\\\ 9\\end\{bmatrix\}\.

In the permuted variant \(right\), the task is significantly more difficult because the features are no longer spatially aligned across columns\. The learner must simultaneously discover the latent rules and the implicit feature correspondence – recognizing, for example, that the constant feature corresponds to the bottom element of column 1, the top element of column 2, and the bottom element of column 3\. After disentangling these mappings and applying the latent rules, the learner must predict the appropriately permuted final panel,\[93\]\\begin\{bmatrix\}9\\\\ 3\\end\{bmatrix\}\.

#### Task structure

Each task is structured as a3×33\\times 3grid of panels, where each panel contains anFF\-dimensional feature vector of integers, governed by a hidden combination ofFFrules and ordered according to a column\-specific permutation\. The rules operate horizontally, such that the features of the third column in each row are determined by applying independent rules to the features in the first two columns\. The same sequence of rules is applied consistently across all three rows\. The available rules encompass constant patterns, arithmetic progressions, modular addition or subtraction, minimum/maximum operations, and the distribution of distinct elements\.

#### Support and query set

To generate a task, we sample a hidden combination of rules, one for each feature dimension as well as column\-specific feature permutations\. The first two rows of the3×33\\times 3matrix act as the support set, demonstrating the applied rules, while the first two columns of the third row serve as the query\.

#### Task invariances

The column\-specific feature permutations can be used to create isomorphic transformations of the same task, rebinding the hidden features of the unpermuted task to the observed feature orderings \(feature rebinding\)\.

### 4\.3Program induction over integer sequences

Next, we consider rule induction over integer sequences as studied by[Rule et al\., \(2024\)](https://arxiv.org/html/2609.13948#bib.bib43)in which a learner must infer a latent rule to transform an input list of integers into an output list of integers\. Since the original set of tasks were handcrafted, we define a probabilistic context\-free grammar that allows us to procedurally sample a large number of similar tasks and systematically create isomorphic task variations\.

#### Instructions

We provide the general task description shown at the top of Figure[13](https://arxiv.org/html/2609.13948#A3.F13)to the learner in the main evaluation and perform additional experiments where we additionally describe the basic operations from which each task is composed in the instructions as shown at the bottom of Figure[13](https://arxiv.org/html/2609.13948#A3.F13)\.

#### Example

The following is a sample episode from the program induction over integer sequences task family\.

\[1, 2, 3, 4\]→\[4, 1, 2\]\\displaystyle\\rightarrow\\texttt\{\[4, 1, 2\]\}\[1, 8, 3\]→\[3, 1\]\\displaystyle\\rightarrow\\texttt\{\[3, 1\]\}\[8, 9, 1, 2, 4\]→\[2, 4, 8, 9\]\\displaystyle\\rightarrow\\texttt\{\[2, 4, 8, 9\]\}\[0, 1\]→\[0\]\\displaystyle\\rightarrow\\texttt\{\[0\]\}\[3, 3, 4, 5, 5, 6\]→\[5, 6, 3, 3, 4\]\\displaystyle\\rightarrow\\texttt\{\[5, 6, 3, 3, 4\]\}\[2\]→\[\]\\displaystyle\\rightarrow\\texttt\{\[\]\}\[7, 3, 1, 5, 2\]→?\\displaystyle\\rightarrow\\textbf\{?\}To predict the query in this example, the learner must infer that the target rule is a composition of two operations: first swapping the two halves of the list, and then removing the element at the first position\. It then has to apply these operations to the novel test input sequence to predict the answer, here\[5, 2, 7, 3\]\.

#### Task structure

The underlying program for each task is sampled from a probabilistic context\-free grammar that allows to create compositions of operations that take integer lists as input and output integer lists\. The grammar supports parameterized operations such asinsertorremove, which take additional integer arguments such as structural positions, sequence lengths, or vocabulary elements\. The full list of operations is shown at the bottom of Figure[13](https://arxiv.org/html/2609.13948#A3.F13)\. By composing these operations, the generative model produces a single deterministic expression that parses and transforms any given input list\. Parameters and input sequence integers are sampled uniformly from predefined ranges\.

#### Support and query set

We randomly sample input lists of integers in the range of zero to nine for the support and query set, strictly partitioning the sequence lengths of the support and query sets into disjoint pools\. Specifically, we ensure that the support set contains lists that are both shorter and longer than the lists in the query set\. The query set examples are subsequently sampled from intermediate sequence lengths that are never seen in the support set\. This is to prevent the existence of a shortcut solution in which – rather than inferring the underlying program – it would be possible to simply copy the analogical elements of the same length example\.

#### Task invariances

We can create isomorphic task variations by permuting the order of the support and query samples \(example reordering\)\. In addition, we can apply a bijective mapping to the integer vocabulary of the elements in the support and query set as well as implicitly to any integer parameters in the hidden rule, e\.g\., the value argument inserted by theinsertoperation \(symbol substitution\)\.

### 4\.4Boolean concept learning

Finally, we adapt the Boolean concept learning task family studied in[Feldman, \(2000\)](https://arxiv.org/html/2609.13948#bib.bib15)and[Piantadosi et al\., \(2016\)](https://arxiv.org/html/2609.13948#bib.bib39)\. In each task, a learner must infer a latent concept from examples to classify objects with varying physical features, e\.g\. shapes with different colors and sizes into binary categories\.

#### Instructions

Figure[14](https://arxiv.org/html/2609.13948#A3.F14)shows the general task instructions we provide in the system prompt asking models to identify thesimplestrule to classify the support examples\. What constitutes thesimplestrule is underspecified without knowledge of the description language\. For this reason, for each task we describe the context\-free grammar from which the rule can be generated and with respect to which it is the shortest rule that explains the support set, see Figure[14](https://arxiv.org/html/2609.13948#A3.F14)for an example\.

#### Example

The following is a sample episode from the Boolean concept learning task family\.

→wudsy\\displaystyle\\rightarrow\\texttt\{wudsy\}→not wudsy\\displaystyle\\rightarrow\\texttt\{not wudsy\}→wudsy\\displaystyle\\rightarrow\\texttt\{wudsy\}→?\\displaystyle\\rightarrow\\textbf\{?\}To predict the query example of this task, the learner must infer that the simplest Boolean rule that explains the support set is thatall red objects are wudsy\.

wudsynot wudsyFigure 5:Example Hamming graph used to construct the support set from objects at the decision boundary \(dashed line\) of a given rule\.
#### Task structure

We procedurally generate Boolean logic expressions of varying complexity from a probabilistic context\-free grammar\. Our grammar supports generating compositions of the logical operatorsandandor, as well as base predicates that evaluate specific features of the objects \(e\.g\.,is\_color\(x, red\)\. Notably, the grammar does not contain anotoperator to reduce redundancy in the hypothesis space over possible rules\. The particular features objects have in a specific task are a random subset of three out of five possible features \(shape, size, color, material, and numerosity\), where each feature can take three distinct values \(e\.g\., red, blue, green\)\.

#### Support and query set

When naively sampling a support set for a given rule, it is likely that a simpler rule than the one used to label the support examples can equally explain the support set\. To contend with this possibility, we strategically build the support set and use an ideal observer model to ensure the rule used to generate the set is unique and the simplest to explain it\. Specifically, to create the support set, we construct the Hamming graph over all theoretically possible objects, with an edge between any two objects that differ by exactly one feature as shown in Figure[5](https://arxiv.org/html/2609.13948#S4.F5)\. As a heuristic, we can then select all pairs of adjacent objects in the Hamming graph that lie on the decision boundary of the given rule and randomly sample the query set from the remaining objects\.

#### Task invariances

The specific physical feature to which a rule applies should not matter to a systematic learner, i\.e\. the ruleall red objects are wudsyis structurally equivalent toall squares are wudsy\. We can therefore create isomorphic task variations both by permuting the order of the support and query samples \(example reordering\) as well as by rebinding the physical features of a rule and correspondingly changing the support and query set \(symbol substitution\)\.

## 5Evaluating reasoning models across systematic task variations

We now evaluate systematicity of current state\-of\-the\-art proprietary reasoning models on the rule induction task families introduced above\. For each task, we generate a set of isomorphic task variations, compute the systematicity metrics defined in Section[3\.5](https://arxiv.org/html/2609.13948#S3.SS5)and report the average across task variation sets within a task family\. While ideally we would cover all possible variations, there is typically a prohibitively large number of possible variations\. Throughout our experiments, we therefore sampleK=5K=5variations per task isomorphism\.

Apart from experiments where we explicitly vary the reasoning effort, we evaluate all models with reasoning effort set to ‘high’ and use the default sampling parameters of the official APIs for each model\. We use structured outputs to ensure models output consistently formatted responses according to the schema requirements of each task family\. We compare models from three major model providers in the main evaluation and focus on Gemini models to examine the effect of reasoning effort to keep costs reasonable\.

### 5\.1Models can solve some task variations but struggle to systematically solve all variations

Figure[1](https://arxiv.org/html/2609.13948#S0.F1)shows average systematicity metrics over task variation sets on the four rule induction task families\. Across most of the task families, we find that there is a notable systematicity gap, i\.e\. a clear difference between the fraction of task variation sets where any variation was solved and the fraction of task variation sets where all variations were solved\. TheBoolean category learningtask is a notable exception to this observation\. Almost all models can solve all tasks and variations thereof, demonstrating that perfect systematicity is achievable\.

The systematicity gap appears most clearly forsymbolic Raven’s progressive matricesandprogram induction over integer sequences\. For example, in91\.67%91\.67\\%of the task variation sets in the program induction task family, Claude Opus 4\.7 can solve at least one variation but only in50\.00%50\.00\\%of the task variation sets can it solve all variations, leaving a systematicity gap of41\.67%41\.67\\%\. GPT\-5\.5

We can decrease the difficulty of the program induction tasks by providing a detailedhintof the possible program components \(see Figure[13](https://arxiv.org/html/2609.13948#A3.F13)\) and accordingly observe a notable improvement in performance for stronger models in Figure[9](https://arxiv.org/html/2609.13948#A2.F9)albeit without fully resolving the systematicity gap\. The Raven’s matrices task family already uses the easiest parameterization with only two features\. Making it more difficult by increasing the number of features to three predictably further decreases model performance further, as shown in Figure[10](https://arxiv.org/html/2609.13948#A2.F10)\.

On thegrammar\-based instruction\-learningtask family overall performance is modest and models lack systematicity with the notable exception of GPT\-5\.5: if it can solve any task variation it can solve all of them, otherwise it solves none\. Figure[8](https://arxiv.org/html/2609.13948#A2.F8)A further demonstrates that the systematicity gap is not specific to a particular type of task isomorphism but appears to varying degrees for recomposition, reordering and substitution\. In addition, Figure[8](https://arxiv.org/html/2609.13948#A2.F8)B shows that while adding extra support examples alongside the curriculum examples makes the task easier, the systematicity gap remains\.

Figure 6:Reasoning effort does not consistently improve systematicity\.Varying the reasoning effort for different Gemini reasoning models across task families does not consistently improve systematicity\.
### 5\.2Reasoning effort does not consistently improve systematicity

Inference\-time scaling with increased reasoning budgets typically improves model performance\. While our main comparison in Figure[1](https://arxiv.org/html/2609.13948#S0.F1)was conducted with reasoning effort set to high, we would like to understand the impact of the reasoning effort on systematicity\. Apriori, one might expect that increasing the reasoning effort could reduce the impact of task variations on model variance and therefore possibly improve systematicity\.

Figure[6](https://arxiv.org/html/2609.13948#S5.F6)shows the systematicity metrics across task families for Gemini models when varying the reasoning effort fromlowtomedium, andhigh\. Surprisingly, increasing reasoning effort does not consistently improve systematicity\. On theprogram induction over integer sequencesandsymbolic Raven’s progressive matricestask families, we can observe a modest improvement in task performance but the systematicity gap largely remains unaffected\. On thegrammar\-based instruction\-learningtask family the effect is highly model\-dependent\.

### 5\.3Model stochasticity only partially explains the systematicity gap

Since reasoning models rely on stochastic sampling, two sources of variance affect model systematicity: the within\-task\-variation sampling noise and the between\-task\-variation variance due to the isomorphic task variations\. In the following, we try to delineate the impact of either, first by decomposing the empirically observed variances and then by re\-evaluating models with zero temperature sampling to eliminate sampling noise \(up to numerical stability\)\.

#### Large variance for multiple attempts on the same task\.

By applying the law of total variance and using suitable empirical estimators, we can decompose the total variance across task variations and attempts for a given task into the within\-task\-variation variance and the between\-task\-variation variance\. In Appendix[A](https://arxiv.org/html/2609.13948#A1), we formally state this decomposition and develop the respective estimators\. Table[1](https://arxiv.org/html/2609.13948#A1.T1)lists the different variance components across task families, type of task variation and model\. We find that within\-task\-variation variance often makes up the largest part of the total variance implying that large sampling noise dominates \(or rather masks as we will see in the next section\) between\-task\-variation variance\.

#### Systematicity gap persists with greedy decoding

Figure 7:Systematicity gap persists with greedy decoding\.When setting the decoding sampling temperature to zero during inference, reasoning model task performance remains comparable and the systematicity gap persists\.By setting the sampling temperature to zero we can, in principle, eliminate the within\-task\-variation variance\. We re\-evaluate Gemini models111Neither GPT\-5\.5 nor Claude Opus 4\.7 support zero temperature reasoning, so we have to limit this evaluation to Gemini\.in this quasi\-deterministic setting222Floating point arithmetic can lead to residual stochasticity during greedy decoding\., finding that the systematicity gap and model performance remains largely unchanged, as shown in Figure[7](https://arxiv.org/html/2609.13948#S5.F7)\. This suggests that the lack of systematicity observed in reasoning models goes beyond the noisy nature of stochastic decoding\.

## 6Discussion

We investigated systematicity of thought of reasoning models on four rule induction task families – grammar\-based instruction\-learning, symbolic reasoning, program induction over integer sequences and Boolean concept learning\. Based on the compositional structure of each task family, we identify task isomorphisms that allow us to generate structurally equivalent task variations\. Across most task families we find that reasoning models lack strict systematicity of thought: despite being able to solve a task, they often fail to solve structurally equivalent variants of it\. These results demonstrate the difficulty of robustly establishing the cognitive abilities of reasoning models beyond the particular contexts they are evaluated in\. Our findings further raise the following points of discussion:

#### Reliability is a prerequisite for systematicity\.

Consistent behavior when repeatedly facing the exact same context is a requirement for systematicity\. Our findings reveal that even with majority voting over five independent attempts, current reasoning models often do not satisfy this requirement\.

#### Machines should strive to be more systematic than humans\.

We can evaluate systematicity in reasoning models in its strictest form by comparing independent behavioral outcomes across task variations given the exact same initial conditions\. This is not possible with humans where any experience will alter future behavioral outcomes\. Nevertheless, humans are arguably not perfectly systematic either: Many variables such as mood, tiredness or semantics can affect performance despite having the competence to solve a task in principle\([Chomsky,, 1965](https://arxiv.org/html/2609.13948#bib.bib11);[Wason,, 1968](https://arxiv.org/html/2609.13948#bib.bib50);[Evans et al\.,, 1983](https://arxiv.org/html/2609.13948#bib.bib14);[Ashby et al\.,, 1999](https://arxiv.org/html/2609.13948#bib.bib1);[Van Der Linden et al\.,, 2003](https://arxiv.org/html/2609.13948#bib.bib47)\)\. In comparison, the intelligent machines we build are tireless and we should strive for them to be as systematic as possible\.

#### Test validity requires systematicity\.

A lack of systematicity raises a fundamental issue of test validity in the evaluation of reasoning models: Tests have to commit to a particular context and therefore differ by definition from the plethora of contexts within which a system is expected to operate\. Without systematicity, we lack a strong reason to expect that the ability to do something in one context is predictive of this ability in other contexts\.

#### Limitations

Our evaluation relies on synthetic data generation in order to create the controlled, isomorphic task variations needed to isolate systematicity\. Since measuring systematicity requires evaluating many reasoning chains per task, it is costly and we limit the number of task variations we generate per task toK=5K=5\(each of which is evaluated overN=5N=5attempts to compute the majority vote\)\. Ultimately, a strictly systematic system should solve most task variations, regardless of whether it is evaluated onK=5K=5orK=100K=100variations\.

#### Broader impacts

This paper evaluates the systematicity of existing reasoning models\. While we foresee no immediate negative societal impact, we hope that it may improve our understanding of this widely deployed technology\.

#### Acknowledgements

We would like to thank Ryan Burnell, Philipp Mondorf, Changho Shin and Solim LeGris for insightful discussions and valuable feedback\. Simon Schug is supported by Postdoc\.Mobility grantP500PT\_225369from the Swiss National Science Foundation\. Brenden M\. Lake is supported by the U\.S\. National Science Foundation \(NSF\) under Cooperative Agreement No\. 2433429, NSF AI Research Institute on Interaction for AI Assistants \(ARIA\)\. We thank Google for generously supporting this research with a financial gift and by providing Google Cloud Credits to evaluate Gemini models\.

## References

- Ashby et al\., \(1999\)Ashby, F\. G\., Isen, A\. M\., and Turken, A\. U\. \(1999\)\.A neuropsychological theory of positive affect and its influence on cognition\.Psychological Review, 106\(3\):529–550\.
- Berglund et al\., \(2024\)Berglund, L\., Tong, M\., Kaufmann, M\., Balesni, M\., Stickland, A\., Korbak, T\., and Evans, O\. \(2024\)\.The Reversal Curse: LLMs trained on ”A is B” fail to learn ”B is A”\.InI Can’t Believe It’s Not Better Workshop: Failure Modes in the Age of Foundation Models\.
- Biewald, \(2020\)Biewald, L\. \(2020\)\.Experiment Tracking with Weights and Biases\.
- Boole, \(1854\)Boole, G\. \(1854\)\.An investigation of the laws of thought,\.New York, Dover\.
- Bradbury et al\., \(2018\)Bradbury, J\., Frostig, R\., Hawkins, P\., Johnson, M\. J\., Leary, C\., Maclaurin, D\., Necula, G\., Paszke, A\., VanderPlas, J\., Wanderman\-Milne, S\., and Zhang, Q\. \(2018\)\.JAX: composable transformations of Python\+NumPy programs\.
- Bubeck et al\., \(2023\)Bubeck, S\., Chandrasekaran, V\., Eldan, R\., Gehrke, J\., Horvitz, E\., Kamar, E\., Lee, P\., Lee, Y\. T\., Li, Y\., Lundberg, S\., Nori, H\., Palangi, H\., Ribeiro, M\. T\., and Zhang, Y\. \(2023\)\.Sparks of Artificial General Intelligence: Early experiments with GPT\-4\.arXiv:2303\.12712 \[cs\]\.
- Burnell et al\., \(2026\)Burnell, R\., Yamamori, Y\., Firat, O\., Olszewska, K\., Hughes\-Fitt, S\., Kelly, O\., Galatzer\-Levy, I\. R\., Morris, M\. R\., Dafoe, A\., Snyder, A\. M\., Goodman, N\. D\., Botvinick, M\., and Legg, S\. \(2026\)\.Measuring Progress Toward AGI: A Cognitive Framework\.arXiv:2605\.28405 \[cs\.AI\] version: 1\.
- Carpenter et al\., \(1990\)Carpenter, P\. A\., Just, M\. A\., and Shell, P\. \(1990\)\.What one intelligence test measures: a theoretical account of the processing in the Raven Progressive Matrices Test\.Psychological Review, 97\(3\):404–431\.
- Chen et al\., \(2025\)Chen, Y\.\-H\., Davidson, G\., and Lake, B\. \(2025\)\.SAGE\-Eval: Evaluating LLMs for Systematic Generalizations of Safety Facts\.In Belgrave, D\., Zhang, C\., Lin, H\., Pascanu, R\., Koniusz, P\., Ghassemi, M\., and Chen, N\., editors,Advances in Neural Information Processing Systems, volume 38\. Curran Associates, Inc\.
- Cho et al\., \(2025\)Cho, S\., Ruberto, S\., and Terragni, V\. \(2025\)\.Metamorphic Testing of Large Language Models for Natural Language Processing\.In2025 IEEE International Conference on Software Maintenance and Evolution \(ICSME\), pages 174–186\.ISSN: 2576\-3148\.
- Chomsky, \(1965\)Chomsky, N\. \(1965\)\.Aspects of the Theory of Syntax\.MIT Press, Cambridge\.
- Chomsky, \(1985\)Chomsky, N\. \(1985\)\.Syntactic structures\.Number 4 in Janua Linguarum Series minor\. Mouton Publ, The Hague, 14\. printing edition\.
- Elazar et al\., \(2021\)Elazar, Y\., Kassner, N\., Ravfogel, S\., Ravichander, A\., Hovy, E\., Schütze, H\., and Goldberg, Y\. \(2021\)\.Measuring and Improving Consistency in Pretrained Language Models\.Transactions of the Association for Computational Linguistics, 9:1012–1031\.
- Evans et al\., \(1983\)Evans, J\. S\. B\. T\., Barston, J\. L\., and Pollard, P\. \(1983\)\.On the conflict between logic and belief in syllogistic reasoning\.Memory & Cognition, 11\(3\):295–306\.
- Feldman, \(2000\)Feldman, J\. \(2000\)\.Minimization of Boolean complexity in human concept learning\.Nature, 407\(6804\):630–633\.
- Fodor, \(1975\)Fodor, J\. \(1975\)\.The Language of Thought\.Language and thought series\. Harvard University Press\.
- Fodor and Pylyshyn, \(1988\)Fodor, J\. A\. and Pylyshyn, Z\. W\. \(1988\)\.Connectionism and cognitive architecture: A critical analysis\.Cognition, 28\(1\-2\):3–71\.
- Goodman et al\., \(2015\)Goodman, N\. D\., Tenenbaum, J\. B\., and Gerstenberg, T\. \(2015\)\.Concepts in a probabilistic language of thought\.InThe Conceptual Mind: New Directions in the Study of Concepts, pages 623–653\. MIT Press\.
- Griffiths, \(2026\)Griffiths, T\. \(2026\)\.The Laws of Thought: The Quest for a Mathematical Theory of the Mind\.HarperCollins Publishers\.
- Guo et al\., \(2025\)Guo, D\., Yang, D\., Zhang, H\., Song, J\., Wang, P\., Zhu, Q\., Xu, R\., Zhang, R\., Ma, S\., Bi, X\., Zhang, X\., Yu, X\., Wu, Y\., Wu, Z\. F\., Gou, Z\., Shao, Z\., Li, Z\., Gao, Z\., Liu, A\., Xue, B\., Wang, B\., Wu, B\., Feng, B\., Lu, C\., Zhao, C\., Deng, C\., Ruan, C\., Dai, D\., Chen, D\., Ji, D\., Li, E\., Lin, F\., Dai, F\., Luo, F\., Hao, G\., Chen, G\., Li, G\., Zhang, H\., Xu, H\., Ding, H\., Gao, H\., Qu, H\., Li, H\., Guo, J\., Li, J\., Chen, J\., Yuan, J\., Tu, J\., Qiu, J\., Li, J\., Cai, J\. L\., Ni, J\., Liang, J\., Chen, J\., Dong, K\., Hu, K\., You, K\., Gao, K\., Guan, K\., Huang, K\., Yu, K\., Wang, L\., Zhang, L\., Zhao, L\., Wang, L\., Zhang, L\., Xu, L\., Xia, L\., Zhang, M\., Zhang, M\., Tang, M\., Zhou, M\., Li, M\., Wang, M\., Li, M\., Tian, N\., Huang, P\., Zhang, P\., Wang, Q\., Chen, Q\., Du, Q\., Ge, R\., Zhang, R\., Pan, R\., Wang, R\., Chen, R\. J\., Jin, R\. L\., Chen, R\., Lu, S\., Zhou, S\., Chen, S\., Ye, S\., Wang, S\., Yu, S\., Zhou, S\., Pan, S\., Li, S\. S\., Zhou, S\., Wu, S\., Yun, T\., Pei, T\., Sun, T\., Wang, T\., Zeng, W\., Liu, W\., Liang, W\., Gao, W\., Yu, W\., Zhang, W\., Xiao, W\. L\., An, W\., Liu, X\., Wang, X\., Chen, X\., Nie, X\., Cheng, X\., Liu, X\., Xie, X\., Liu, X\., Yang, X\., Li, X\., Su, X\., Lin, X\., Li, X\. Q\., Jin, X\., Shen, X\., Chen, X\., Sun, X\., Wang, X\., Song, X\., Zhou, X\., Wang, X\., Shan, X\., Li, Y\. K\., Wang, Y\. Q\., Wei, Y\. X\., Zhang, Y\., Xu, Y\., Li, Y\., Zhao, Y\., Sun, Y\., Wang, Y\., Yu, Y\., Zhang, Y\., Shi, Y\., Xiong, Y\., He, Y\., Piao, Y\., Wang, Y\., Tan, Y\., Ma, Y\., Liu, Y\., Guo, Y\., Ou, Y\., Wang, Y\., Gong, Y\., Zou, Y\., He, Y\., Xiong, Y\., Luo, Y\., You, Y\., Liu, Y\., Zhou, Y\., Zhu, Y\. X\., Huang, Y\., Li, Y\., Zheng, Y\., Zhu, Y\., Ma, Y\., Tang, Y\., Zha, Y\., Yan, Y\., Ren, Z\. Z\., Ren, Z\., Sha, Z\., Fu, Z\., Xu, Z\., Xie, Z\., Zhang, Z\., Hao, Z\., Ma, Z\., Yan, Z\., Wu, Z\., Gu, Z\., Zhu, Z\., Liu, Z\., Li, Z\., Xie, Z\., Song, Z\., Pan, Z\., Huang, Z\., Xu, Z\., Zhang, Z\., and Zhang, Z\. \(2025\)\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature, 645\(8081\):633–638\.
- Harris et al\., \(2020\)Harris, C\. R\., Millman, K\. J\., Walt, S\. J\. v\. d\., Gommers, R\., Virtanen, P\., Cournapeau, D\., Wieser, E\., Taylor, J\., Berg, S\., Smith, N\. J\., Kern, R\., Picus, M\., Hoyer, S\., Kerkwijk, M\. H\. v\., Brett, M\., Haldane, A\., Río, J\. F\. d\., Wiebe, M\., Peterson, P\., Gérard\-Marchant, P\., Sheppard, K\., Reddy, T\., Weckesser, W\., Abbasi, H\., Gohlke, C\., and Oliphant, T\. E\. \(2020\)\.Array programming with NumPy\.Nature, 585\(7825\):357–362\.
- Hupkes et al\., \(2020\)Hupkes, D\., Dankers, V\., Mul, M\., and Bruni, E\. \(2020\)\.Compositionality Decomposed: How do Neural Networks Generalise?Journal of Artificial Intelligence Research, 67:757–795\.
- Inc, \(2015\)Inc, P\. T\. \(2015\)\.Collaborative data science\.Place: Montreal, QC\.
- Jang et al\., \(2022\)Jang, M\., Kwon, D\. S\., and Lukasiewicz, T\. \(2022\)\.BECEL: Benchmark for Consistency Evaluation of Language Models\.In Calzolari, N\., Huang, C\.\-R\., Kim, H\., Pustejovsky, J\., Wanner, L\., Choi, K\.\-S\., Ryu, P\.\-M\., Chen, H\.\-H\., Donatelli, L\., Ji, H\., Kurohashi, S\., Paggio, P\., Xue, N\., Kim, S\., Hahm, Y\., He, Z\., Lee, T\. K\., Santus, E\., Bond, F\., and Na, S\.\-H\., editors,Proceedings of the 29th International Conference on Computational Linguistics, pages 3680–3696, Gyeongju, Republic of Korea\. International Committee on Computational Linguistics\.
- Jang and Lukasiewicz, \(2023\)Jang, M\. and Lukasiewicz, T\. \(2023\)\.Consistency Analysis of ChatGPT\.In Bouamor, H\., Pino, J\., and Bali, K\., editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15970–15985, Singapore\. Association for Computational Linguistics\.
- Jia and Liang, \(2017\)Jia, R\. and Liang, P\. \(2017\)\.Adversarial Examples for Evaluating Reading Comprehension Systems\.In Palmer, M\., Hwa, R\., and Riedel, S\., editors,Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark\. Association for Computational Linguistics\.
- Jiang et al\., \(2020\)Jiang, Z\., Xu, F\. F\., Araki, J\., and Neubig, G\. \(2020\)\.How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 8:423–438\.
- Kaplan et al\., \(2020\)Kaplan, J\., McCandlish, S\., Henighan, T\., Brown, T\. B\., Chess, B\., Child, R\., Gray, S\., Radford, A\., Wu, J\., and Amodei, D\. \(2020\)\.Scaling Laws for Neural Language Models\.arXiv:2001\.08361 \[cs\]\.
- Lake and Baroni, \(2023\)Lake, B\. M\. and Baroni, M\. \(2023\)\.Human\-like systematic generalization through a meta\-learning neural network\.Nature, 623\(7985\):115–121\.
- Lake et al\., \(2017\)Lake, B\. M\., Ullman, T\. D\., Tenenbaum, J\. B\., and Gershman, S\. J\. \(2017\)\.Building machines that learn and think like people\.Behavioral and Brain Sciences, 40:e253\.
- Lightman et al\., \(2024\)Lightman, H\., Kosaraju, V\., Burda, Y\., Edwards, H\., Baker, B\., Lee, T\., Leike, J\., Schulman, J\., Sutskever, I\., and Cobbe, K\. \(2024\)\.Let’ s Verify Step by Step\.In Kim, B\., Yue, Y\., Chaudhuri, S\., Fragkiadaki, K\., Khan, M\., and Sun, Y\., editors,International Conference on Learning Representations, volume 2024, pages 39578–39601\.
- Lu et al\., \(2022\)Lu, Y\., Bartolo, M\., Moore, A\., Riedel, S\., and Stenetorp, P\. \(2022\)\.Fantastically Ordered Prompts and Where to Find Them: Overcoming Few\-Shot Prompt Order Sensitivity\.In Muresan, S\., Nakov, P\., and Villavicencio, A\., editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 8086–8098, Dublin, Ireland\. Association for Computational Linguistics\.
- Marsh, \(2024\)Marsh, C\. \(2024\)\.uv: An extremely fast Python package and project manager, written in Rust\.
- McLaughlin, \(1993\)McLaughlin, B\. P\. \(1993\)\.The connectionism/classicism battle to win souls\.Philosophical Studies, 71\(2\):163–190\.
- Mondorf et al\., \(2026\)Mondorf, P\., Bell, S\. J\., Dodge, J\., and Hupkes, D\. \(2026\)\.LPDS: Evaluating LLM Robustness Through Logic\-Preserving Difficulty Scaling\.\_eprint: 2605\.15393\.
- OpenAI et al\., \(2024\)OpenAI, Jaech, A\., Kalai, A\., Lerer, A\., Richardson, A\., El\-Kishky, A\., Low, A\., Helyar, A\., Madry, A\., Beutel, A\., Carney, A\., Iftimie, A\., Karpenko, A\., Passos, A\. T\., Neitz, A\., Prokofiev, A\., Wei, A\., Tam, A\., Bennett, A\., Kumar, A\., Saraiva, A\., Vallone, A\., Duberstein, A\., Kondrich, A\., Mishchenko, A\., Applebaum, A\., Jiang, A\., Nair, A\., Zoph, B\., Ghorbani, B\., Zhang, B\., Rossen, B\., Sokolowsky, B\., Barak, B\., McGrew, B\., Minaiev, B\., Hao, B\., Baker, B\., Houghton, B\., McKinzie, B\., Eastman, B\., Lugaresi, C\., Bassin, C\., Hudson, C\., Li, C\. M\., Bourcy, C\. d\., Voss, C\., Shen, C\., Zhang, C\., Koch, C\., Orsinger, C\., Hesse, C\., Fischer, C\., Chan, C\., Roberts, D\., Kappler, D\., Levy, D\., Selsam, D\., Dohan, D\., Farhi, D\., Mely, D\., Robinson, D\., Tsipras, D\., Li, D\., Oprica, D\., Freeman, E\., Zhang, E\., Wong, E\., Proehl, E\., Cheung, E\., Mitchell, E\., Wallace, E\., Ritter, E\., Mays, E\., Wang, F\., Such, F\. P\., Raso, F\., Leoni, F\., Tsimpourlas, F\., Song, F\., Lohmann, F\. v\., Sulit, F\., Salmon, G\., Parascandolo, G\., Chabot, G\., Zhao, G\., Brockman, G\., Leclerc, G\., Salman, H\., Bao, H\., Sheng, H\., Andrin, H\., Bagherinezhad, H\., Ren, H\., Lightman, H\., Chung, H\. W\., Kivlichan, I\., O’Connell, I\., Osband, I\., Gilaberte, I\. C\., Akkaya, I\., Kostrikov, I\., Sutskever, I\., Kofman, I\., Pachocki, J\., Lennon, J\., Wei, J\., Harb, J\., Twore, J\., Feng, J\., Yu, J\., Weng, J\., Tang, J\., Yu, J\., Candela, J\. Q\., Palermo, J\., Parish, J\., Heidecke, J\., Hallman, J\., Rizzo, J\., Gordon, J\., Uesato, J\., Ward, J\., Huizinga, J\., Wang, J\., Chen, K\., Xiao, K\., Singhal, K\., Nguyen, K\., Cobbe, K\., Shi, K\., Wood, K\., Rimbach, K\., Gu\-Lemberg, K\., Liu, K\., Lu, K\., Stone, K\., Yu, K\., Ahmad, L\., Yang, L\., Liu, L\., Maksin, L\., Ho, L\., Fedus, L\., Weng, L\., Li, L\., McCallum, L\., Held, L\., Kuhn, L\., Kondraciuk, L\., Kaiser, L\., Metz, L\., Boyd, M\., Trebacz, M\., Joglekar, M\., Chen, M\., Tintor, M\., Meyer, M\., Jones, M\., Kaufer, M\., Schwarzer, M\., Shah, M\., Yatbaz, M\., Guan, M\. Y\., Xu, M\., Yan, M\., Glaese, M\., Chen, M\., Lampe, M\., Malek, M\., Wang, M\., Fradin, M\., McClay, M\., Pavlov, M\., Wang, M\., Wang, M\., Murati, M\., Bavarian, M\., Rohaninejad, M\., McAleese, N\., Chowdhury, N\., Chowdhury, N\., Ryder, N\., Tezak, N\., Brown, N\., Nachum, O\., Boiko, O\., Murk, O\., Watkins, O\., Chao, P\., Ashbourne, P\., Izmailov, P\., Zhokhov, P\., Dias, R\., Arora, R\., Lin, R\., Lopes, R\. G\., Gaon, R\., Miyara, R\., Leike, R\., Hwang, R\., Garg, R\., Brown, R\., James, R\., Shu, R\., Cheu, R\., Greene, R\., Jain, S\., Altman, S\., Toizer, S\., Toyer, S\., Miserendino, S\., Agarwal, S\., Hernandez, S\., Baker, S\., McKinney, S\., Yan, S\., Zhao, S\., Hu, S\., Santurkar, S\., Chaudhuri, S\. R\., Zhang, S\., Fu, S\., Papay, S\., Lin, S\., Balaji, S\., Sanjeev, S\., Sidor, S\., Broda, T\., Clark, A\., Wang, T\., Gordon, T\., Sanders, T\., Patwardhan, T\., Sottiaux, T\., Degry, T\., Dimson, T\., Zheng, T\., Garipov, T\., Stasi, T\., Bansal, T\., Creech, T\., Peterson, T\., Eloundou, T\., Qi, V\., Kosaraju, V\., Monaco, V\., Pong, V\., Fomenko, V\., Zheng, W\., Zhou, W\., Zhan, W\., McCabe, W\., Zaremba, W\., Dubois, Y\., Lu, Y\., Chen, Y\., Cha, Y\., Bai, Y\., He, Y\., Zhang, Y\., Wang, Y\., Shao, Z\., and Li, Z\. \(2024\)\.OpenAI o1 System Card\.Technical report\.\_eprint: 2412\.16720\.
- Ouyang et al\., \(2022\)Ouyang, L\., Wu, J\., Jiang, X\., Almeida, D\., Wainwright, C\., Mishkin, P\., Zhang, C\., Agarwal, S\., Slama, K\., Ray, A\., Schulman, J\., Hilton, J\., Kelton, F\., Miller, L\., Simens, M\., Askell, A\., Welinder, P\., Christiano, P\. F\., Leike, J\., and Lowe, R\. \(2022\)\.Training language models to follow instructions with human feedback\.In Koyejo, S\., Mohamed, S\., Agarwal, A\., Belgrave, D\., Cho, K\., and Oh, A\., editors,Advances in Neural Information Processing Systems, volume 35, pages 27730–27744\. Curran Associates, Inc\.
- Penn et al\., \(2008\)Penn, D\. C\., Holyoak, K\. J\., and Povinelli, D\. J\. \(2008\)\.Darwin’s mistake: Explaining the discontinuity between human and nonhuman minds\.Behavioral and Brain Sciences, 31\(2\):109–130\.
- Piantadosi et al\., \(2016\)Piantadosi, S\. T\., Tenenbaum, J\. B\., and Goodman, N\. D\. \(2016\)\.The logical primitives of thought: Empirical foundations for compositional cognitive models\.Psychological Review, 123\(4\):392–424\.
- Raven, \(1962\)Raven, J\. C\. \(1962\)\.Advanced Progressive Matrices, Set II\.H\. K\. Lewis, London\.
- Ribeiro et al\., \(2020\)Ribeiro, M\. T\., Wu, T\., Guestrin, C\., and Singh, S\. \(2020\)\.Beyond Accuracy: Behavioral Testing of NLP Models with CheckList\.In Jurafsky, D\., Chai, J\., Schluter, N\., and Tetreault, J\., editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online\. Association for Computational Linguistics\.
- Romanou et al\., \(2026\)Romanou, A\., Ibrahim, M\., Ross, C\., Shaib, C\., Oktar, K\., Bell, S\. J\., Ovalle, A\., Dodge, J\., Bosselut, A\., Sinha, K\., and Williams, A\. \(2026\)\.Brittlebench: Quantifying LLM robustness via prompt sensitivity\.\_eprint: 2603\.13285\.
- Rule et al\., \(2024\)Rule, J\. S\., Piantadosi, S\. T\., Cropper, A\., Ellis, K\., Nye, M\., and Tenenbaum, J\. B\. \(2024\)\.Symbolic metaprogram search improves learning efficiency and explains rule learning in humans\.Nature Communications, 15\(1\):6847\.
- Schug, \(2025\)Schug, S\. \(2025\)\.Meta\-Learning & Compositional Generalization in Neural Networks\.PhD thesis, ETH Zurich\.
- Schug et al\., \(2025\)Schug, S\., Kobayashi, S\., Akram, Y\., Sacramento, J\., and Pascanu, R\. \(2025\)\.Attention as a Hypernetwork\.In Yue, Y\., Garg, A\., Peng, N\., Sha, F\., and Yu, R\., editors,International Conference on Learning Representations, volume 2025, pages 68744–68770\.
- Segura et al\., \(2016\)Segura, S\., Fraser, G\., Sanchez, A\. B\., and Ruiz\-Cortés, A\. \(2016\)\.A Survey on Metamorphic Testing\.IEEE Transactions on Software Engineering, 42\(9\):805–824\.
- Van Der Linden et al\., \(2003\)Van Der Linden, D\., Frese, M\., and Meijman, T\. F\. \(2003\)\.Mental fatigue and the control of cognitive processes: effects on perseveration and planning\.Acta Psychologica, 113\(1\):45–65\.
- Wang et al\., \(2023\)Wang, B\., Chen, W\., Pei, H\., Xie, C\., Kang, M\., Zhang, C\., Xu, C\., Xiong, Z\., Dutta, R\., Schaeffer, R\., Truong, S\., Arora, S\., Mazeika, M\., Hendrycks, D\., Lin, Z\., Cheng, Y\., Koyejo, S\., Song, D\., and Li, B\. \(2023\)\.DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models\.In Oh, A\., Naumann, T\., Globerson, A\., Saenko, K\., Hardt, M\., and Levine, S\., editors,Advances in Neural Information Processing Systems, volume 36, pages 31232–31339\. Curran Associates, Inc\.
- Wang et al\., \(2022\)Wang, X\., Wei, J\., Schuurmans, D\., Le, Q\. V\., Chi, E\. H\., Narang, S\., Chowdhery, A\., and Zhou, D\. \(2022\)\.Self\-Consistency Improves Chain of Thought Reasoning in Language Models\.
- Wason, \(1968\)Wason, P\. C\. \(1968\)\.Reasoning about a Rule\.Quarterly Journal of Experimental Psychology, 20\(3\):273–281\.
- Wei et al\., \(2023\)Wei, A\., Haghtalab, N\., and Steinhardt, J\. \(2023\)\.Jailbroken: How Does LLM Safety Training Fail?In Oh, A\., Naumann, T\., Globerson, A\., Saenko, K\., Hardt, M\., and Levine, S\., editors,Advances in Neural Information Processing Systems, volume 36, pages 80079–80110\. Curran Associates, Inc\.
- Wei et al\., \(2022\)Wei, J\., Wang, X\., Schuurmans, D\., Bosma, M\., Ichter, B\., Xia, F\., Chi, E\. H\., Le, Q\. V\., and Zhou, D\. \(2022\)\.Chain\-of\-thought prompting elicits reasoning in large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 24824–24837, Red Hook, NY, USA\. Curran Associates Inc\.
- Zhao et al\., \(2021\)Zhao, Z\., Wallace, E\., Feng, S\., Klein, D\., and Singh, S\. \(2021\)\.Calibrate Before Use: Improving Few\-shot Performance of Language Models\.In Meila, M\. and Zhang, T\., editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 12697–12706\. PMLR\.
- Zou et al\., \(2023\)Zou, A\., Wang, Z\., Carlini, N\., Nasr, M\., Kolter, J\. Z\., and Fredrikson, M\. \(2023\)\.Universal and Transferable Adversarial Attacks on Aligned Language Models\.arXiv:2307\.15043 \[cs\]\.

## Appendix ATask variance decomposition

For each task we sampleK\>1K\>1task variations and evaluate a model’s binary successN\>1N\>1times on each task variation\. This leaves us with two sources of variance: the within\-task\-variation sampling noise and the between\-task\-variation variance due to the isomorphic task variations\. Both of these sources can ultimately make the models appear unsystematic\. We quantify their contribution to the total variance in the following\.

### A\.1Theoretical quantities

For a single fixed base taskTT, we define the following random variables:

- •Letτ\\taube a random variable representing the task variationτ⁡\(T\)\\tau\(T\)\.
- •Letp^τ\\hat\{p\}\_\{\\tau\}be the empirical success rate of the model on task variationτ\\tau, calculated overNNindependent attempts\.
- •Letpτ:=𝔼⁡\[p^τ\|τ\]p\_\{\\tau\}:=\\E\[\\hat\{p\}\_\{\\tau\}\|\\tau\]be the true underlying success probability of the model on task variationτ\\tau\.

To understand the total variance in the observed success rates,p^τ\\hat\{p\}\_\{\\tau\}, across all possible task variations, we apply the law of total variance:

𝕍​ar⁡\(p^τ\)\\displaystyle\\Var\(\\hat\{p\}\_\{\\tau\}\)=𝔼τ⁡\[𝕍​ar⁡\(p^τ\|τ\)\]\+𝕍​arτ⁡\(𝔼⁡\[p^τ\|τ\]\)\\displaystyle=\\E\_\{\\tau\}\[\\Var\(\\hat\{p\}\_\{\\tau\}\|\\tau\)\]\+\\Var\_\{\\tau\}\(\\E\[\\hat\{p\}\_\{\\tau\}\|\\tau\]\)\(1\)
Becausep^τ\\hat\{p\}\_\{\\tau\}is the sample mean ofNNindependent Bernoulli trials, each with success probabilitypτp\_\{\\tau\}, the variance of the empirical success rate given the task variation is𝕍​ar⁡\(p^τ\|τ\)=pτ​\(1−pτ\)N\\Var\(\\hat\{p\}\_\{\\tau\}\|\\tau\)=\\frac\{p\_\{\\tau\}\(1\-p\_\{\\tau\}\)\}\{N\}\. Substituting Equation \([1](https://arxiv.org/html/2609.13948#A1.E1)\) and rearranging the terms yields:

𝕍​arτ⁡\(pτ\)\\displaystyle\\Var\_\{\\tau\}\(p\_\{\\tau\}\)=𝕍​ar⁡\(p^τ\)−𝔼τ⁡\[pτ​\(1−pτ\)N\]\.\\displaystyle=\\Var\(\\hat\{p\}\_\{\\tau\}\)\-\\E\_\{\\tau\}\\left\[\\frac\{p\_\{\\tau\}\(1\-p\_\{\\tau\}\)\}\{N\}\\right\]\.This equation allows us to isolate the true between\-task\-variation variance of the underlying success probabilities,𝕍​arτ⁡\(pτ\)\\Var\_\{\\tau\}\(p\_\{\\tau\}\), from the expected within\-task\-variation variance purely due to the finite sampling noise ofNNattempts per task variation,𝔼τ⁡\[pτ​\(1−pτ\)N\]\\E\_\{\\tau\}\\left\[\\frac\{p\_\{\\tau\}\(1\-p\_\{\\tau\}\)\}\{N\}\\right\]using the total variation of the empirical success rate,𝕍​ar⁡\(p^τ\)\\Var\(\\hat\{p\}\_\{\\tau\}\)\.

### A\.2Empirical estimates

For a given task, we sampleKKtask variations\. Letp^τk\\hat\{p\}\_\{\\tau\_\{k\}\}be the model’s empirical success rate for sampled variationτk\\tau\_\{k\}\(wherek∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}\), andp^\\hat\{p\}be the model’s overall empirical success rate for this task across allK⋅NK\\cdot Nattempts\.

We estimate the between\-task\-variation variance,σ^between2\\hat\{\\sigma\}^\{2\}\_\{\\text\{between\}\}, of the underlying success probabilities across variations by removing the estimated within\-task\-variation variance,σ^within2\\hat\{\\sigma\}^\{2\}\_\{\\text\{within\}\}from the estimated total variance,σ^total2\\hat\{\\sigma\}^\{2\}\_\{\\text\{total\}\}, as outlined in the derivation above:

σ^between2\\displaystyle\\hat\{\\sigma\}^\{2\}\_\{\\text\{between\}\}=σ^total2−σ^within2\\displaystyle=\\hat\{\\sigma\}^\{2\}\_\{\\text\{total\}\}\-\\hat\{\\sigma\}^\{2\}\_\{\\text\{within\}\}
We compute the following empirical estimators for the variance components:

#### Total observed variance

We estimate the sample variance of the variation success rates as:

σ^total2\\displaystyle\\hat\{\\sigma\}^\{2\}\_\{\\text\{total\}\}=1K−1​∑k=1K\(p^τk−p^\)2\\displaystyle=\\frac\{1\}\{K\-1\}\\sum\_\{k=1\}^\{K\}\(\\hat\{p\}\_\{\\tau\_\{k\}\}\-\\hat\{p\}\)^\{2\}

#### Expected within\-task\-variation variance

To prevent true variation differences from inflating the estimated noise, we use the following unbiased local estimator to calculate the expected binomial variance overNNattempts,

σ^within2\\displaystyle\\hat\{\\sigma\}^\{2\}\_\{\\text\{within\}\}=1K​∑k=1Kp^τk​\(1−p^τk\)N−1\.\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\frac\{\\hat\{p\}\_\{\\tau\_\{k\}\}\(1\-\\hat\{p\}\_\{\\tau\_\{k\}\}\)\}\{N\-1\}\.

#### Normalization

Since the empirical success rates differ between tasks, so do the theoretically obtainable maximum variances, which makes a comparison difficult\. For this reason, we normalize the three variance components, expressing them as a proportion of the maximum possible variance\. With eachp^τk∈\[0,1\]\\hat\{p\}\_\{\\tau\_\{k\}\}\\in\[0,1\], the total observed sample variance is bounded by a theoretical maximum ofσm​a​x2=KK−1​p^​\(1−p^\)\\sigma^\{2\}\_\{max\}=\\frac\{K\}\{K\-1\}\\hat\{p\}\(1\-\\hat\{p\}\)\. We then obtain the

- •Normalized total variance,ρtotal=σ^total2σmax2\\rho\_\{\\text\{total\}\}=\\frac\{\\hat\{\\sigma\}^\{2\}\_\{\\text\{total\}\}\}\{\\sigma^\{2\}\_\{\\text\{max\}\}\},
- •Normalized within\-task\-variation variance,ρwithin=σ^within2σmax2\\rho\_\{\\text\{within\}\}=\\frac\{\\hat\{\\sigma\}^\{2\}\_\{\\text\{within\}\}\}\{\\sigma^\{2\}\_\{\\text\{max\}\}\},
- •Normalized between\-task\-variation variance,ρbetween=σ^between2σmax2\\rho\_\{\\text\{between\}\}=\\frac\{\\hat\{\\sigma\}^\{2\}\_\{\\text\{between\}\}\}\{\\sigma^\{2\}\_\{\\text\{max\}\}\}\.

### A\.3Results

In Table[1](https://arxiv.org/html/2609.13948#A1.T1), we report the average normalized variance components across task families, type of task variation and model\. We find that the within\-task\-variation variance often makes up the largest part of the total variance\.

Table 1:Variance decomposition across tasks and models\.Average estimated variance components for each model and task family grouped by the specific task variation type\. Since our estimators are noisy,ρwithin\\rho\_\{\\text\{within\}\}can be larger thanρtotal\\rho\_\{\\text\{total\}\}, in which case we clipρbetween\\rho\_\{\\text\{between\}\}to zero\.

## Appendix BAdditional results

Figure[8](https://arxiv.org/html/2609.13948#A2.F8)shows additional results on the grammar\-based instruction learning task family, Figure[9](https://arxiv.org/html/2609.13948#A2.F9)shows additional results on the program induction over integer sequences task family, and Figure[10](https://arxiv.org/html/2609.13948#A2.F10)shows additional results on the Symbolic Raven’s progressive matrices task family, all described in Section[5](https://arxiv.org/html/2609.13948#S5)of the main text\.

Figure 8:Grammar\-based instruction\-learning\.\(A\) The systematicity gap persists across variation type\. \(B\) Showing 8 rather than 4 support examples in addition to the curriculum examples increases the average number of tasks where any variation is solved but does not resolve the systematicity gap\.Figure 9:Program induction over integer sequences\.Providing hints in the task instructions improves performance for Gemini 3\.1 Flash and Gemini 3\.1 Pro and almost eliminates the systematicity gap in the latter\.Figure 10:Symbolic Raven’s progressive matrices\.Increasing the number of features notably increases difficulty and leads to a corresponding drop in the average number of tasks where any variation can be solved\.
## Appendix CTask instructions

For each task family, we provide detailed instructions in the system prompt\. Figure[11](https://arxiv.org/html/2609.13948#A3.F11)lists the instructions for the grammar\-based instruction\-learning task family, Figure[12](https://arxiv.org/html/2609.13948#A3.F12)lists the instructions for the symbolic Raven’s progressive matrices task family, and Figure[13](https://arxiv.org/html/2609.13948#A3.F13)lists the instructions for the program induction over integer sequences task family\.

Your task is to discover the grammatical rules of an artificial language\. In the following, you will be presented with examples of input and output string pairs\. These pairs demonstrate how sequences of primitive tokens are transformed by grammatical rules\. Afterwards, you will be presented with n\_query new input sequence\(s\), and you have to translate them into output sequences based on the grammatical rules you have inferred\. Try to identify the simplest deterministic rules that govern these transformations and apply them to the new input sequences\.Every artificial language consists of:1\.Primitive rules: Simple one\-to\-one mappings from an input token to an output token\.2\.Function rules: These operate on one or two arguments\. Functions are represented by specific tokens and apply to their adjacent arguments\.•A 1\-argument function appears after its argument \(e\.g\., "arg1 func"\)\. A 2\-argument function appears between its arguments \(e\.g\., "arg1 func arg2"\)\.•Each argument is either a single primitive token or a sequence of tokens\.•A function transforms its evaluated arguments into a new sequence by duplicating and/or rearranging them\.•When the same 2\-argument function is chained with itself, it evaluates left\-to\-right \(e\.g\., "arg1 func arg2 func arg3" is evaluated as "\[arg1 func arg2\] func arg3"\) otherwise functions that accept sequences follow a strict priority order: If Rule A evaluates before Rule B, and Rule B before Rule C, then Rule A always evaluates before Rule C\.Figure 11:Instructions for the grammar\-based instruction\-learning task family\.Your task is to predict integers in structured sequences of integers\. Specifically, you will be presented with tasks that consist of three rows with three columns each\. Each panel in this three by three matrix contains n\_features feature\(s\) encoded as integer\(s\)\. Within a row the n\_features feature\(s\) of the last column can be predicted by applying the correct rules to the first two columns\. Within a task, all rows follow the same underlying rules applied to different inputs\.Each row is delimited by \|\|, each column is delimited by \| and the features within a panel are separated by spaces\. The last n\_features feature\(s\) in the last panel are masked with n\_features question marks, ’?’\.1\.Try to identify which integers in a given task are governed by the same underlying rules\.2\.Try to identify the n\_features rule\(s\) that govern a task\.3\.Consider the following rules/patterns:•Constant, e\.g\. the same integer is repeated•Progression, e\.g\. integers are systematically incremented or decremented by steps of \+1, \+2, \-1, or \-2•Modular addition, e\.g\. two integers prior in the sequence are summed to produce the next integer•Modular subtraction, e\.g\. two integers prior in the sequence are subtracted to produce the next integer•Maximum/Minimum, e\.g\. the maximum/minimum of two integers prior in the sequence makes up the next integerFigure 12:Instructions for the symbolic Raven’s progressive matrices task family\.Your task is to discover the rule that transforms input lists of integers into output lists of integers\. In the following, you will be presented with n\_support examples of input and output list pairs\. Afterwards, you will be presented with n\_query new input list\(s\) and you have to apply the inferred rule to predict their corresponding output lists\.

With hints

The rule is a composition of the following basic operations:•insert: Insert an element at a specific position\.•remove: Remove an element at a specific position\.•repeat: Repeat the list until it reaches a specified length\.•shift: Shift all elements by a given offset, wrapping around\.•swap: Swap the first and second halves of the list\.•tail: Keep only a specified number of elements from the end of the list\.

Figure 13:Instructions for the program induction over integer sequences task family\.Your task is to discover the meaning of ’wudsy’, a word in an alien language\. In the following you will be presented with several examples of objects that are either wudsy or not wudsy\. Afterwards you will be presented with n\_query new object\(s\) and you have to determine whether they are either wudsy or not wudsy\. Try to identify the simplest, deterministic rule for what makes objects wudsy or not wudsy and apply it to decide for the new objects\.Each rule is generated from a context\-free grammar\. The specific grammar and properties of the objects will be provided for each task\. The shortest generated rule that explains the examples is the correct one to base the decision on\.Note in particular that this grammar does not contain a not\(\) operation, keep this in mind when trying to identify the "simplest" rule, i\.e\. the shortest rule that explains all examples that can be expressed in this grammar\.Respond with the boolean value for each of the n\_query new object\(s\) in their presented order according to the rule you identified\.

Example task\-specific instruction

Each object has a shape, color and size\. The rules are generated from the following context\-free grammar:S\\displaystyle S→’true’​∣’false’​∣and​\(S,S\)​∣or​\(S,S\)∣​is\_shape​\(O,A\)∣​is\_color​\(O,B\)∣​is\_size​\(O,C\)\\displaystyle\\to\\texttt\{'true'\}\\mid\\texttt\{'false'\}\\mid\\texttt\{and\}\(S,S\)\\mid\\texttt\{or\}\(S,S\)\\mid\\texttt\{is\\\_shape\}\(O,A\)\\mid\\texttt\{is\\\_color\}\(O,B\)\\mid\\texttt\{is\\\_size\}\(O,C\)O\\displaystyle O→’x’\\displaystyle\\to\\texttt\{'x'\}A\\displaystyle A→’circle’​∣’triangle’∣​’square’\\displaystyle\\to\\texttt\{'circle'\}\\mid\\texttt\{'triangle'\}\\mid\\texttt\{'square'\}B\\displaystyle B→’blue’​∣’yellow’∣​’green’\\displaystyle\\to\\texttt\{'blue'\}\\mid\\texttt\{'yellow'\}\\mid\\texttt\{'green'\}C\\displaystyle C→’tiny’​∣’small’∣​’large’\\displaystyle\\to\\texttt\{'tiny'\}\\mid\\texttt\{'small'\}\\mid\\texttt\{'large'\}

Figure 14:Instructions for the Boolean concept learning task family\.
## Appendix DAdditional details

### D\.1Software and libraries

For the results obtained in this paper we build on free and open\-source software\. We implemented our experiments in Python using NumPy\([Harris et al\.,, 2020](https://arxiv.org/html/2609.13948#bib.bib21)\)and JAX\([Bradbury et al\.,, 2018](https://arxiv.org/html/2609.13948#bib.bib5), Apache License 2\.0\)\. We run API models using Mozilla’s any\-llm library \(Apache License 2\.0\), utilized WandB\([Biewald,, 2020](https://arxiv.org/html/2609.13948#bib.bib3), MIT license\)to monitor the progress and results of experiments, and Plotly\([Inc,, 2015](https://arxiv.org/html/2609.13948#bib.bib23), MIT license\)for generating the plots\. We use uv for Python project dependency management\([Marsh,, 2024](https://arxiv.org/html/2609.13948#bib.bib33), MIT License\)\.