auto-psych: Automating the science of mind using agent-driven theory discovery and experimentation

# Automating the science of mind using agent-driven theory discovery and experimentation
Source: [https://arxiv.org/html/2606.26460](https://arxiv.org/html/2606.26460)
\[Scale=0\.85\]

Kushin MukherjeeDepartment of Psychology, Stanford UniversityDaniel WurgaftDepartment of Psychology, Stanford UniversityLinas NasvytisDepartment of Psychology, Stanford UniversityMichael Y\. LiDepartment of Computer Science, Stanford UniversityNoah D\. GoodmanDepartment of Psychology, Stanford UniversityDepartment of Computer Science, Stanford UniversityMichael C\. FrankDepartment of Psychology, Stanford University

###### Abstract

AI\-based scientific automation is increasingly possible by using agents to generate hypotheses, design experiments, and analyze data\. Data collection is a major bottleneck in this pipeline, however\. Psychology, and computational cognitive science in particular, is well\-positioned to benefit from AI experimentation because theories are often represented as code and crowdsourcing platforms enable programmatic human data collection at scale\. Here, we apply automated discovery techniques to the project of generating theories in computational cognitive science, with an agent\-based system collecting human data independently through crowdsourced survey experiments\. As a testbed, we use a classic case study from cognitive psychology: judging which sequences of coin flips seem subjectively more random\. Our system,auto\-psych, uses nested agent\-based discovery loops to generate explanatory theories of human behavior\. The inner loop conjectures, fits, and critiques probabilistic cognitive models; the outer loop designs experiments to test these models, launches them online, and analyzes the data\. This system can quickly and reliably recover ground\-truth theories from synthetic data via systematic experimentation, but the nested structure is critical to model performance\. Further, in three independent sequences of human experiments, the system finds theories that fit the data better than theories generated from the scientific literature\. This work thus demonstrates the feasibility of automated data collection and theory discovery in computational cognitive science\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.26460v1/x1.png)Figure 1:An agent\-based discovery loop for automated cognitive science\.In the*outer loop*, an LLM agent designs an experiment by proposing candidate stimuli, which are then ranked by their expected information gain over a*model registry*\. Selected stimuli are implemented as a jsPsych experiment and deployed to participants on Prolific\. The collected human data are passed to the*inner loop*\. In the inner loop, each registry model is first fit to the data\. A critic agent then performs posterior\-predictive checks: it both proposes and computes test statistics on datasets simulated from the best incumbent model and compares this reference distribution to the value computed on the human data\. A theorist agent uses this critique to propose new probabilistic models\. The best\-fitting model is carried forward to the model registry\. In practice, we have an additional theorist agent in the outer loop proposing models before designing experiments on iterations after the first\.The scientific process is often described as an iterative loop in which researchers conjecture a theory, devise specific experimental tests of that theory, and then adjust their theory in response to evidence\(Godfrey\-Smith,[2009](https://arxiv.org/html/2606.26460#bib.bib30)\)\.111This idealized characterization may hold at the level of scientists developing explanations of individual phenomena, rather than describing progress in scientific fields as a whole\(Kuhn,[1962](https://arxiv.org/html/2606.26460#bib.bib68)\)\.AI tools can already assist with each step in this loop – helping to create theories, design experiments, and analyze and interpret data\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib55); Jagadishet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib35); Davieset al\.,[2021](https://arxiv.org/html/2606.26460#bib.bib56); Jumperet al\.,[2021](https://arxiv.org/html/2606.26460#bib.bib57); Liet al\.,[2024a](https://arxiv.org/html/2606.26460#bib.bib33); Schmidgallet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib67); Gandhiet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib54)\)\. Could AI systems complete every part of the discovery loop? Such an advance could lead to increasing automation of the scientific process and faster progress towards important applications\(Musslicket al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib1); Jagadishet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib35)\)\. Agent\-based systems built on large language models can now use computational tools and integrate with data collection platforms, making this goal increasingly feasible\. As a step towards it, the current paper develops a framework that automates the complete model discovery loop in computational cognitive science\.

Recent progress has been made in developing language model\-based workflows across the sciences, but these projects have focused on model discovery from existing datasets\. The pattern of iterative model testing and refinement on a known dataset is sometimes known as “Box’s loop” \(after statistician George Box\)\(Blei,[2014](https://arxiv.org/html/2606.26460#bib.bib28)\)\. Because no new data are needed, language model agents can quickly propose, fit, and critique models\(Liet al\.,[2024a](https://arxiv.org/html/2606.26460#bib.bib33);[b](https://arxiv.org/html/2606.26460#bib.bib32); Gandhiet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib54)\)\. However, a critical part of the scientific process is collectingnewdata under targeted experimental regimes to test theories\. Thus, current AI\-driven science requires a “lab\-in\-the\-loop” model\(Swansonet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib27); Ghareebet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib22)\), which limits the speed of iteration because of the need for scientists to carry out the experiments\.

End\-to\-end scientific automation has been a major goal in machine learning research, where AI agents can run experiments*without*humans in the loop\. For example, one automated system for computational research decomposes each problem into a set of steps – idea generation, novelty checking, experimentation, and paper writing\(Luet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib36)\); parallel passes through this workflow with comparison and pruning at each step resulted in an agent\-generated paper that passed human review\(cf\. Beelet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib12)\)\. Even forin silicosystems, reducing data collection bottlenecks is critical to rapid iteration\(Karpathy,[2026](https://arxiv.org/html/2606.26460#bib.bib25)\)\.

Experimentation in psychology requires human participants, but many studies are run online via crowdsourcing platforms like Amazon Mechanical Turk and Prolific\(Buhrmesteret al\.,[2018](https://arxiv.org/html/2606.26460#bib.bib47); Palan and Schitter,[2018](https://arxiv.org/html/2606.26460#bib.bib51)\)\. Such experiments are typically served in template\-based web frameworks\(De Leeuw,[2015](https://arxiv.org/html/2606.26460#bib.bib40)\), meaning that they can be created by artificial agents and deployed via application programming interfaces \(APIs\)\. Just as autonomous machine learning research depends on small\-scale training runs, autonomous psychology can make use of fast and inexpensive online behavioral experiments\.

Computational cognitive science is a sub\-field of psychology that might have the most to gain from automated experimentation\. Extensive work has been done to formalize theories in this domain\. In particular, probabilistic models in the Bayesian tradition provide a unifying language for describing contentful hypotheses about the mind\(Griffithset al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib52)\)\. These theories can be expressed in a number of highly expressive and well\-documented probabilistic programming languages \(e\.g\., Stan, PyMC, Pyro\)\(Goodman,[2013](https://arxiv.org/html/2606.26460#bib.bib43); Binghamet al\.,[2019](https://arxiv.org/html/2606.26460#bib.bib50); Carpenteret al\.,[2017](https://arxiv.org/html/2606.26460#bib.bib49); Patilet al\.,[2010](https://arxiv.org/html/2606.26460#bib.bib48)\), meaning that they can be generated by coding agents and compared with one another using standard statistical approaches\. While previous work has automatically generated and compared cognitive models of this type\(Rmuset al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib34)\), their workflows have yet to be integrated into a full human experimentation loop\.

In the current paper, we implement an automated theory discovery and experimentation loop for computational cognitive science \(Figure[1](https://arxiv.org/html/2606.26460#S1.F1)\), which we callauto\-psych\. We start with a simple testbed: a well\-studied problem in the psychology literature – what makes a sequence of coin flips appear more or less random\(Bar\-Hillel and Wagenaar,[1991](https://arxiv.org/html/2606.26460#bib.bib24); Ayton and Fischer,[2004](https://arxiv.org/html/2606.26460#bib.bib23); Kahneman and Tversky,[1972](https://arxiv.org/html/2606.26460#bib.bib18); Griffiths and Tenenbaum,[2003](https://arxiv.org/html/2606.26460#bib.bib38); Griffithset al\.,[2018](https://arxiv.org/html/2606.26460#bib.bib39)\)\. Subjective randomness is a compelling psychological problem because we have strong intuitions that, say,HTHTHTHTis a*less*random sequence thanHTTHHHTH, despite the two being equiprobable for a fair coin\. Using this problem as our case study, we develop an agent\-based scientific discovery framework for creating computational theories\. The core of this system is two nested loops: an outer loop that proposes probabilistic cognitive models, designs online experiments, and collects and analyzes data \(the “lab\-in\-the\-loop”\), and an inner loop that critiques cognitive models and iteratively refines them \(Box’s loop\)\.

We show that our system reliably recovers ground\-truth models \(behavioral proxies\) through iterative experimentation\. We then let our agents run chains of experiments on human participants, finding strong theory discovery\. In the discussion, we reflect on the promise of this system as well as potential downsides of autonomous scientific agents\.

## 2Related work

Much recent development has focused on autonomous computational discovery, distinguishing single\-agent systems\(Karpathy,[2026](https://arxiv.org/html/2606.26460#bib.bib25); Jianget al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib6)\)from multi\-agent systems that coordinate ensembles of agents with different roles\(Gaoet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib7); Swansonet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib27)\)\(an approach that we follow here\)\. Work in both of these traditions has developed methods for automatically generating, critiquing, and revising statistical models\(Liet al\.,[2024a](https://arxiv.org/html/2606.26460#bib.bib33);[b](https://arxiv.org/html/2606.26460#bib.bib32); Éltetőet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib64); Agarwalet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib63)\)as well as benchmarking their experiment planning abilities\(Konet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib5)\)\.

Similar approaches have yielded positive results in structural biology, helping predict protein structures and biochemical interactions on par within vitroexperimentation, enabling the rapid development of novel theories\(Gottweiset al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib19); Ghareebet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib22); Swansonet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib27); Jumperet al\.,[2021](https://arxiv.org/html/2606.26460#bib.bib57); M\. Branet al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib61); Jinet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib65); Gaoet al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib66)\)\. Agent\-based workflows have found solutions to open problems and proposed new algorithms in mathematics\(Novikovet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib60); Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib55); Fawziet al\.,[2022](https://arxiv.org/html/2606.26460#bib.bib59)\), and have been applied successfully to the social sciences and cognitive science\(Ballaet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib8); Rmuset al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib34); Jagadishet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib35); Genget al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib62); Éltetőet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib64)\)\. Of this work,Rmuset al\.\([2025](https://arxiv.org/html/2606.26460#bib.bib34)\)is closest to our work as they use AI models as part of a pipeline to propose and validate probabilistic cognitive models\. However, none of these works has sought to automate the entire process of computational cognitive science, from conjecturing and evaluating hypotheses to designing and running experiments with real human participants\.

Our specific work here focuses on subjective randomness – people’s perception and judgments of how random a given sequence of events is\. A substantial literature has considered heuristics that people might use\(Bar\-Hillel and Wagenaar,[1991](https://arxiv.org/html/2606.26460#bib.bib24); Ayton and Fischer,[2004](https://arxiv.org/html/2606.26460#bib.bib23)\)\. Popular accounts include representativeness of a prototype\(Kahneman and Tversky,[1972](https://arxiv.org/html/2606.26460#bib.bib18)\), algorithmic complexity\(Falk and Konold,[1997](https://arxiv.org/html/2606.26460#bib.bib69); Li and Vitányi,[2008](https://arxiv.org/html/2606.26460#bib.bib44)\), Bayesian inference accounts\(Griffiths and Tenenbaum,[2003](https://arxiv.org/html/2606.26460#bib.bib38); Griffithset al\.,[2018](https://arxiv.org/html/2606.26460#bib.bib39)\), and accounts that appeal to a finite attentional window\(Hahn and Warren,[2009](https://arxiv.org/html/2606.26460#bib.bib17)\)\.

## 3Methods

Our framework uses LLM\-based agents in coding harnesses to instantiate two loops: an inner loop that conjectures, fits, and critiques models and an outer loop that designs and runs experiments\. Both loops use the[OpenCode](https://opencode.ai/)harness withgemini\-3\.1\-pro\-previewas the language model\. We begin by describing the subjective randomness domain, then turn to the details of the workflow\. Code and data are available on[GitHub](https://github.com/mcfrank/auto-psych)\. Further methodological details can be found in Appendix[A](https://arxiv.org/html/2606.26460#A1)\.

### 3\.1Subjective randomness

The primary question in studies of subjective randomness is which sequences of coin flips are perceived to be more random\. For purposes of our study, we choose a simple forced\-choice paradigm in which participants are presented with two sequences and asked to choose between them\. This setting has many desirable properties for automatically generating and testing cognitive models: the space of possible stimuli is discrete and tractable to enumerate, the experimental paradigm is straightforward, and models need only generate a probability to be assigned to each sequence\.

To seed our loop with models from the literature, we added four models: 1\) an “encoding compressibility” model that uses heuristic features like periodicity and long runs as proxies for the compressibility of a sequence\(Falk and Konold,[1997](https://arxiv.org/html/2606.26460#bib.bib69)\), 2\) a simplified version of a “Bayesian diagnosticity” account that approximates the log\-likelihood ratio that a sequence was generated by a fair coin compared to a more regular computational process\(Griffiths and Tenenbaum,[2003](https://arxiv.org/html/2606.26460#bib.bib38); Griffithset al\.,[2018](https://arxiv.org/html/2606.26460#bib.bib39)\), 3\) a “window typicality” model inspired byHahn and Warren \([2009](https://arxiv.org/html/2606.26460#bib.bib17)\)that judges a sequence based on whether its longest run is typical of a fair coin viewed through a finite memory window, and 4\) a “prototype similarity” model that judges sequences’ randomness based on how well their heads/tails imbalance and alternation rate resemble a mental prototype\(Kahneman and Tversky,[1972](https://arxiv.org/html/2606.26460#bib.bib18); Reimerset al\.,[2018](https://arxiv.org/html/2606.26460#bib.bib70)\)\. These models are not identical to the versions developed in the original papers, but they adapt the key ideas from these accounts of subjective randomness judgments\.

### 3\.2Theory framework

In our framework, a cognitive model is instantiated as a probabilistic program: snippets of Python code written in the PyMC framework\(Abril\-Plaet al\.,[2023](https://arxiv.org/html/2606.26460#bib.bib45)\)\. Each program defines a distribution over participant choices as a function of key stimulus properties, for example a sequence’s length, the number of observed Hs, or how often the sequence alternates between H and T\. Each model also contains a choice\-sensitivity parameter that helps determine how strongly a difference in perceived randomness maps to participant response choices and typically includes a bias parameter that captures a left vs\. right response bias\. We place weak priors over these free parameters and infer distributions by fitting the models to observed data\.

### 3\.3Modeling loop

Theauto\-psychworkflow consists of two loops: anouter loopthat adds new theories, chooses stimuli that optimally distinguish between models, and implements experiments\. In the first iteration,auto\-psychis initialized with a set ofseed models\. In subsequent iterations, a theorist agent comes up with at least one new theory in a manner inspired by the “hypothesis search” framework: the agent first writes a theory in natural language, then translates it to PyMC code\(Wanget al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib46)\)\. The theorist’s models are added to the registry of models to be compared\. Next, a design agent generates 100 to 300 candidate stimuli for the experiment and scores them by their expected information gain \(EIG\) with respect to the models under consideration\(Ouyanget al\.,[2016](https://arxiv.org/html/2606.26460#bib.bib72); Fosteret al\.,[2019](https://arxiv.org/html/2606.26460#bib.bib71); Gandhiet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib54)\)\. We compute EIG using a uniform prior over models in the registry on each step\. After choosing stimuli, an implementation agent writes an experiment using jsPsych\. We gave the agent a strict template to follow and automatically added our consent form to the experiment code\. The agent deploys the experiment to Firebase and recruits participants using the Prolific API\. It then polls the Prolific study to monitor when it is completed, pulls the data, and continues the loop\. All experiments were approved under Stanford IRB protocol \#20009\. More details about the human experiments can be found in Appendix[A\.3](https://arxiv.org/html/2606.26460#A1.SS3)\.

After collecting data, the workflow proceeds to the inner loop\. This is an implementation of Box’s loop: a model improvement loop that iteratively critiques and refines cognitive models\. Each iteration of this loop starts with a critic agent adapted from the CriticAL framework\(Liet al\.,[2024b](https://arxiv.org/html/2606.26460#bib.bib32)\)\. The critic conjectures eight test statistics designed to surface differences between the posterior of the best\-fitting model and the human data\. The theorist then sees the results of any test statistics that surfaced a significant difference at thep=0\.05p=0\.05threshold and proposes new models to improve upon the best model\. This loop repeats twice, and the best\-fitting model found in this inner loop is added to the model registry\. The inner and outer loop theorists operate differently; details can be found in Appendix[A](https://arxiv.org/html/2606.26460#A1)\.

## 4Results

We first evaluate our setup using its ability to recover different ground\-truth models\. Next, we report results from human experiments launched by the framework\.

### 4\.1Recovering cognitive models

![Refer to caption](https://arxiv.org/html/2606.26460v1/x2.png)

![Refer to caption](https://arxiv.org/html/2606.26460v1/x3.png)

Figure 2:Root mean squared error between held\-out seed models \(top\) and psychologically implausible “alien” models \(bottom\) for each model scoring step\. Orange lines show the fit of the best seed model \(other than the held\-out one\) fit to all of the data\. Vertical gray lines delimit the three experimental rounds\. Error bars show standard error of the mean\.To evaluate model recovery, we first tested how well theauto\-psychworkflow can recover cognitive models when they are held out from the initial set of seed models\. This analysis is analogous in spirit to checking consistency of an estimator: under data generated from a known model, we ask whether the workflow recovers the generating model, which is an important property for the system to satisfy\. We first ran these model recovery experiments by holding out one seed model from the set\. We sampled responses from the held\-out seed model and measured how closely the best model matched the held\-out ground\-truth model\. We also investigated whether the model could recover a priori implausible models that mismatch human psychology\. We designed four “alien” models in which sequences were judged as more random based on having more/fewer heads, longer runs of the same outcome, and more imbalance between heads and tails\.

For each of these experiments, we collected five independent replicates to assess reliability\. We measured how wellauto\-psychrecovered a ground\-truth model by comparing the ground\-truth model’s responses to the responses of the best\-fitting model that the modeling loop found\. We compared the responses using the root mean squared error \(RMSE\) computed over all possible stimulus pairs of length up to eight\.

The best model discovered byauto\-psychconsistently matched the ground\-truth better than the best\-fitting seed model when fit to all the data generated across the experiments \(Figure[2](https://arxiv.org/html/2606.26460#S4.F2)\)\. Interestingly, final recovery accuracy appears higher for the alien models \(bottom row\) than for the held\-out seed models \(top row\)\. This may be because the alien models are quite simple and thus easier to discover\.

Overall, we take these results as evidence thatauto\-psychcan reliably find cognitive models that match the ground\-truth model from which they are generated\. To test whether the nested loop structure played a critical role in this success, we also ran a variant ofauto\-psychin which we ablated the inner loop\. Recovery performance was notably worse; details are reported in Appendix[B](https://arxiv.org/html/2606.26460#A2)\.

### 4\.2Automatically modeling and collecting data from human participants

![Refer to caption](https://arxiv.org/html/2606.26460v1/x4.png)Figure 3:Comparison of the best\-fitting model proposed by an agent to the seed models at the end of each run\. Error bars denote difference standard error\.As a test of whetherauto\-psychcould discover better models than the seed models from the literature, we launched three independent replicates of our discovery loop with human participants\. Each replicate started with the same seed models and ran three experimental rounds with 40 participants in each round\. Experiments took between 25 and 48 minutes to complete data collection, highlighting the possibility of further iteration\. Across the nine experiments, 262 of 288 stimulus pairs \(91%\) were unique, indicating that the same stimulus pairs were not being used across experiments; median sequences chosen were between six and eight flips long \(with eight being the maximum allowable\)\. Participants showed no meaningful side bias \(mean=0\.50, 95% CI \[0\.49, 0\.51\]\) and only a small heads bias \(0\.52 \[0\.51, 0\.54\]\) but they did rate as more random both longer sequences \(0\.57 \[0\.55, 0\.60\]\) and sequences with more alternations \(0\.62 \[0\.60, 0\.64\]\)\.

The fits of the best model and the seed models in each replicate are shown in Figure[3](https://arxiv.org/html/2606.26460#S4.F3)\. Each of the three replicates discovered a model that explained the human data better than the seed models\. Replicate 1 found a model that definitively beat the seed models\. Replicate 2 found two models that were virtually tied and both beat the seed models\. The best agent\-discovered model only slightly outperformed the Bayesian diagnosticity seed in replicate 3\. The models discovered in different replicates look different, but they make similar predictions about behavior\. The maximum pairwise RMSE between any of the four best\-fitting models with parameters fit to human data \(including both discovered in replicate 2\) was 0\.092\.

Table 1:Model comparison on pooled human data\. Win = winning model for one replicate\.Δ\\DeltaELPD is the difference from the best model and SE\(Δ\\Delta\) its standard error\.kk= free parameters; RMSE andR2R^\{2\}computed over per\-stimulus averages\.Finally, we asked how the best\-fitting models from each replicate performed across the data from all replicates\. We fit the seed models and the discovered winners \(including two tied models for one replicate\) to the full dataset of∼\\sim11k trials\. We computed the ELPD\-LOO score as well as per\-stimulus root mean squared error and per\-stimulusR2R^\{2\}\. Discovered models from replicates 1 and 2 were substantially better than the seed models across all datasets, and the model from replicate 3 slightly outperformed the best seed model \(Table[1](https://arxiv.org/html/2606.26460#S4.T1)\)\. The noise ceiling forR2R^\{2\}, computed as the Spearman\-Brown adjusted split\-half reliability, was \.80, thus the best\-fitting model predicted 83% of the explainable variance in the data\. To test whether these results were inflated due to models overfitting to the datasets on which they were developed, we repeated this comparison, refitting each top model to the two datasets that it was not originally developed on and found similar results \(see Appendix[C](https://arxiv.org/html/2606.26460#A3)\)\.

### 4\.3Qualitative analysis of winning models

The best\-fitting model across all experiments was the “Minkowski typicality” model\. This model compares a sequence’s proportion of heads and alternation rate to a prototype of a random sequence of coin flips\. The model judges a sequence as less random the more it differs from the prototype\. The prototypical alternation rate and proportion of heads are free parameters\. Deviations are weighted by apenalty\_powerexponent, which enables the model to interpolate between forgiving large deviations and punishing them disproportionately\. The posterior mean of this parameter was 1\.35, meaning the distance metric was in between Manhattan distance and Euclidean distance\. Replicate 2 found two similar models that were virtually tied in ELPD\. These are both called “evidence accumulation” models\. In these models, each outcome \(or run\) contributes a fixed amount of evidence that a sequence is random, but that evidence is discounted by the sequence’s squared deviation from a prototype\. These models are conceptually similar to the Minkowski typicality model and highly similar to each other: they achieve an RMSE with each other of 0\.029\. Replicate 3 found a variant of the Bayesian diagnosticity model that adds an “artificial balance” penalty that judges sequences as less random when their proportion of heads is too close to 0\.5\.

While these models all achieve strong fits to the data and make similar predictions, they tell different stories about human cognition\. The models discovered in replicates 1 and 2 are all based on comparison to a mental prototype, but the model discovered in replicate 3 is based on Bayesian accounts of subjective randomness\. Random variation in the collected human data and the agent’s generated models can lead different replicates to arrive at quite different models\. The space of possible models is large, and it can be difficult to distinguish between models that have different internal structure but usually agree on predicted behavior\. Perhaps future experiments will distinguish these candidates\.

## 5Discussion

In this paper, we presentedauto\-psych, a framework for automatically generating, testing, and critiquing computational models of cognition\. The core design feature of this framework is the presence of two nested loops, an inner loop that proposes and critiques models, and an outer loop that designs and runs experiments with human participants to differentiate models\. We appliedauto\-psychto the subjective randomness domain and found that it could recover ground\-truth models informed by the literature as well as psychologically alien models; both loops were important in this process\. We used the framework to implement and run experiments with real human participants, finding that it discovered cognitive models that fit human behavior significantly better than the initial seed models\. Because these seed models are inspired by influential theories of the target domain, this result indicates strong discovery ability\. Our framework is, to the best of our knowledge, the first instance of a fully automated iterative psychology loop in which AI agents design experiments and collect real human data\. At a larger scale, a process like this might be able to explore vast spaces of experimental designs autonomously and find better models of human cognition\.

The agent\-discovered models posited novel revisions to our seed models\. For example, the Minkowski typicality model showed that the extent to which people penalize distance from a prototype is better\-characterized by a distance in between Manhattan and Euclidean distance than by either of the two\. That said, the discovered models were largely conservative revisions to the seed models\.auto\-psychdid not produce an entirely new paradigm for subjective randomness judgments; future iterations may discover whether this was because its instructions were too conservative or because the seed models were already \(partially\) correct models of human judgment\.

### 5\.1Limitations

Like other automated scientific discovery systems\(Swansonet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib27); Luet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib36)\), our work here provides a proof\-of\-concept for feasibility using a single case study\. The subjective randomness problem is a classic case study in cognitive science with a surprisingly rich set of theories in the literature\. Nevertheless, the stimulus space is highly restricted, making it simpler to create experiments and choose stimuli than in many commonly used paradigms\. The system we designed is quite general, and LLM\-based coding agents are improving rapidly\. Therefore, we expect that our work here could be applied to other paradigms, but scaling for any given paradigm will be controlled by the unevenness of the theoretical space as well as the richness of the stimulus and experimental design options\.

Our work here also shares the limitations of much prior work in computational cognitive science\. The focus of this field has primarily been on providing parsimonious high\-level descriptions of human cognition\. Most classic investigations focused on schematic stimuli, simplified theories, participant averages rather than individual variation, and measurements of convenience samples from “WEIRD” populations\(Kroupinet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib14)\)\. We adopted this general – intrinsically limited but still powerful – approach here, recognizing that modern work in this field attempts to circumvent these limitations, including through the use of naturalistic stimuli, richer theoretical vocabularies, and models of individual participants\(Carvalho and Lampinen,[2025](https://arxiv.org/html/2606.26460#bib.bib9); Petersonet al\.,[2021](https://arxiv.org/html/2606.26460#bib.bib16); Lee and Webb,[2005](https://arxiv.org/html/2606.26460#bib.bib15); Tauberet al\.,[2017](https://arxiv.org/html/2606.26460#bib.bib10); Fan,[2026](https://arxiv.org/html/2606.26460#bib.bib73)\)\.

An additional limitation of our approach strikes at the core of psychological science: those models discovered byauto\-psychmay fit data better than human\-created models and may even be measurably more parsimonious \(e\.g\., shorter\)\. Yet these models may still fail to provide satisfying explanations for human psychologists\. Whether deeper explanatory virtues can or should be available to automated systems remains to be seen\.

### 5\.2The promise \(and perils\) of automated scientific discovery

A better understanding of human cognition could lead to important advances in education, clinical treatment, and human\-computer interaction\. Scientific automation has serious potential costs, however\. One of these is the threat of increasing homogeneity in scientific theories\(Khosrowi,[2026](https://arxiv.org/html/2606.26460#bib.bib4); Haoet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib20)\)\. If the only theories scientists consider are those proposed by the same set of theorist agents, they might easily overlook good alternatives that are low probability for those agents\. Additionally, if AI\-driven science led to a flood of publications that human scientists had to review, it could further overwhelm an already stretched peer\-review infrastructure\(Messeri and Crockett,[2026](https://arxiv.org/html/2606.26460#bib.bib21)\)\. Finally, automating science removes training opportunities for scientists, potentially leading to “deskilling” that might decrease the set of individuals qualified to judge the outputs of automated discovery systems\(Lenharo,[2026](https://arxiv.org/html/2606.26460#bib.bib13); Shen and Tamkin,[2026](https://arxiv.org/html/2606.26460#bib.bib11)\)\.

Mitigating these issues will require thoughtful integration of discovery systems with existing scientific infrastructure\. Rather than replacing scientists, small\-scale discovery systems such asauto\-psychmight be a starting point for scientists to compare theories across phenomena, seek integrative models, or iterate before exploring more costly physiological or neural measurements\. Software innovations – such as those we use here for creating probabilistic programs or launching web experiments – have routinely raised the level of abstraction at which scientists can work\. We hope that automated discovery tools can provide the next layer of abstraction to accelerate progress in understanding the mind\.

## Acknowledgments

Data collection and agent usage were partially supported by gift funds and a credit grant from Google Inc\.

## References

- O\. Abril\-Pla, V\. Andreani, C\. Carroll, L\. Dong, C\. J\. Fonnesbeck, M\. Kochurov, R\. Kumar, J\. Lao, C\. C\. Luhmann, O\. A\. Martin,et al\.\(2023\)PyMC: a modern, and comprehensive probabilistic programming framework in Python\.PeerJ Computer Science9,pp\. e1516\.Cited by:[§3\.2](https://arxiv.org/html/2606.26460#S3.SS2.p1.1)\.
- D\. Agarwal, B\. P\. Majumder, R\. Adamson, M\. Chakravorty, S\. R\. Gavireddy, A\. Parashar, H\. Surana, B\. D\. Mishra, A\. McCallum, A\. Sabharwal, and P\. Clark \(2025\)AutoDiscovery: open\-ended scientific discovery via Bayesian surprise\.External Links:2507\.00310,[Link](https://arxiv.org/abs/2507.00310)Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- P\. Ayton and I\. Fischer \(2004\)The hot hand fallacy and the gambler’s fallacy: two faces of subjective randomness?\.Memory & cognition32\(8\),pp\. 1369–1378\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p6.1),[§2](https://arxiv.org/html/2606.26460#S2.p3.1)\.
- J\. Balla, S\. Huang, O\. Dugan, R\. Dangovski, and M\. Soljačić \(2025\)AI\-assisted discovery of quantitative and formal models in social science\.Humanities and Social Sciences Communications12\(1\),pp\. 114\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- M\. Bar\-Hillel and W\. A\. Wagenaar \(1991\)The perception of randomness\.Advances in applied mathematics12\(4\),pp\. 428–454\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p6.1),[§2](https://arxiv.org/html/2606.26460#S2.p3.1)\.
- J\. Beel, M\. Kan, and M\. Baumgart \(2025\)Evaluating sakana’s AI scientist: bold claims, mixed results, and a promising future?\.InACM SIGIR Forum,Vol\.59,pp\. 1–20\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p3.1)\.
- E\. Bingham, J\. P\. Chen, M\. Jankowiak, F\. Obermeyer, N\. Pradhan, T\. Karaletsos, R\. Singh, P\. Szerlip, P\. Horsfall, and N\. D\. Goodman \(2019\)Pyro: deep universal probabilistic programming\.Journal of machine learning research20\(28\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1)\.
- D\. M\. Blei \(2014\)Build, compute, critique, repeat: data analysis with latent variable models\.Annual Review of Statistics and Its Application1\(1\),pp\. 203–232\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p2.1)\.
- M\. D\. Buhrmester, S\. Talaifar, and S\. D\. Gosling \(2018\)An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use\.Perspectives on psychological science13\(2\),pp\. 149–154\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p4.1)\.
- B\. Carpenter, A\. Gelman, M\. D\. Hoffman, D\. Lee, B\. Goodrich, M\. Betancourt, M\. Brubaker, J\. Guo, P\. Li, and A\. Riddell \(2017\)Stan: a probabilistic programming language\.Journal of statistical software76,pp\. 1–32\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1)\.
- W\. Carvalho and A\. Lampinen \(2025\)Naturalistic computational cognitive science: towards generalizable models and theories that capture the full range of natural behavior\.arXiv preprint arXiv:2502\.20349\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- A\. Davies, P\. Veličković, L\. Buesing, S\. Blackwell, D\. Zheng, N\. Tomašev, R\. Tanburn, P\. Battaglia, C\. Blundell, A\. Juhász,et al\.\(2021\)Advancing mathematics by guiding human intuition with ai\.Nature600\(7887\),pp\. 70–74\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1)\.
- J\. R\. De Leeuw \(2015\)JsPsych: a JavaScript library for creating behavioral experiments in a web browser\.Behavior research methods47\(1\),pp\. 1–12\.Cited by:[§A\.3](https://arxiv.org/html/2606.26460#A1.SS3.p1.1),[§1](https://arxiv.org/html/2606.26460#S1.p4.1)\.
- N\. Éltető, N\. D\. Daw, K\. L\. Stachenfeld, and K\. J\. Miller \(2026\)ATLAS: active theory learning for automated science\.arXiv preprint arXiv:2606\.12386\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p1.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- R\. Falk and C\. Konold \(1997\)Making sense of randomness: implicit encoding as a basis for judgment\.\.Psychological review104\(2\),pp\. 301\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p3.1),[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- J\. E\. Fan \(2026\)Generative behaviors as key targets for cognitive models\.Current Directions in Psychological Science,pp\. 09637214261416790\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- A\. Fawzi, M\. Balog, A\. Huang, T\. Hubert, B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, F\. J\. R\. Ruiz, J\. Schrittwieser, G\. Swirszcz,et al\.\(2022\)Discovering faster matrix multiplication algorithms with reinforcement learning\.Nature610\(7930\),pp\. 47–53\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- A\. Foster, M\. Jankowiak, E\. Bingham, P\. Horsfall, Y\. W\. Teh, T\. Rainforth, and N\. Goodman \(2019\)Variational Bayesian optimal experimental design\.External Links:1903\.05480Cited by:[§3\.3](https://arxiv.org/html/2606.26460#S3.SS3.p1.1)\.
- K\. Gandhi, M\. Y\. Li, L\. Goodyear, A\. Bhatia, L\. Li, A\. Bhaskar, M\. Zaman, and N\. D\. Goodman \(2025\)BoxingGym: benchmarking progress in automated experimental design and model discovery\.External Links:2501\.01540,[Link](https://arxiv.org/abs/2501.01540)Cited by:[§A\.1](https://arxiv.org/html/2606.26460#A1.SS1.p1.4),[§1](https://arxiv.org/html/2606.26460#S1.p1.1),[§1](https://arxiv.org/html/2606.26460#S1.p2.1),[§3\.3](https://arxiv.org/html/2606.26460#S3.SS3.p1.1)\.
- S\. Gao, A\. Fang, Y\. Huang, V\. Giunchiglia, A\. Noori, J\. R\. Schwarz, Y\. Ektefaie, J\. Kondic, and M\. Zitnik \(2024\)Empowering biomedical discovery with AI agents\.Cell187\(22\),pp\. 6125–6151\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- S\. Gao, A\. Fang, and M\. Zitnik \(2026\)AutoScientists: self\-organizing agent teams for long\-running scientific experimentation\.External Links:2605\.28655,[Link](https://arxiv.org/abs/2605.28655)Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- J\. Geng, H\. Chen, D\. Arumugam, and T\. L\. Griffiths \(2025\)Are large language models reliable AI scientists? assessing reverse\-engineering of black\-box systems\.arXiv preprint arXiv:2505\.17968\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- A\. E\. Ghareeb, B\. Chang, L\. Mitchener, A\. Yiu, C\. J\. Szostkiewicz, D\. Shved, G\. J\. Gyimesi, J\. M\. Laurent, S\. M\. Wright, M\. T\. Razzak,et al\.\(2026\)A multi\-agent system for automating scientific discovery\.Nature,pp\. 1–3\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p2.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- P\. Godfrey\-Smith \(2009\)Theory and reality: an introduction to the philosophy of science\.University of Chicago Press\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1)\.
- N\. D\. Goodman \(2013\)The principles and practice of probabilistic programming\.ACM SIGPLAN Notices48\(1\),pp\. 399–402\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1)\.
- J\. Gottweis, W\. Weng, A\. Daryin, T\. Tu, P\. Sirkovic, A\. Myaskovsky, G\. Glowaty, F\. Weissenberger, A\. Orlandi, D\. Popovici,et al\.\(2026\)Accelerating scientific discovery with co\-scientist\.Nature,pp\. 1–3\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- T\. L\. Griffiths, N\. Chater, and J\. B\. Tenenbaum \(2024\)Bayesian models of cognition: reverse engineering the mind\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1)\.
- T\. L\. Griffiths, D\. Daniels, J\. L\. Austerweil, and J\. B\. Tenenbaum \(2018\)Subjective randomness as statistical inference\.Cognitive psychology103,pp\. 85–109\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p6.1),[§2](https://arxiv.org/html/2606.26460#S2.p3.1),[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- T\. Griffiths and J\. Tenenbaum \(2003\)From algorithmic to subjective randomness\.Advances in neural information processing systems16\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p6.1),[§2](https://arxiv.org/html/2606.26460#S2.p3.1),[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- U\. Hahn and P\. A\. Warren \(2009\)Perceptions of randomness: why three heads are better than four\.\.Psychological review116\(2\),pp\. 454\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p3.1),[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- Q\. Hao, F\. Xu, Y\. Li, and J\. Evans \(2026\)Artificial intelligence tools expand scientists’ impact but contract science’s focus\.Nature,pp\. 1–7\.Cited by:[§5\.2](https://arxiv.org/html/2606.26460#S5.SS2.p1.1)\.
- A\. K\. Jagadish, M\. Rmus, K\. Witte, M\. Mathony, M\. Binz, and E\. Schulz \(2026\)Can we automatize scientific discovery in the cognitive sciences?\.External Links:2603\.20988,[Link](https://arxiv.org/abs/2603.20988)Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- Z\. Jiang, D\. Schmidt, D\. Srikanth, D\. Xu, I\. Kaplan, D\. Jacenko, and Y\. Wu \(2025\)AIDE: ai\-driven exploration in the space of code\.External Links:2502\.13138,[Link](https://arxiv.org/abs/2502.13138)Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- R\. Jin, M\. Xu, F\. Meng, G\. Wan, Q\. Cai, Y\. Jiang, J\. Han, Y\. Chen, W\. Lu, M\. Wang,et al\.\(2025\)STELLA: towards a biomedical world model with self\-evolving multimodal agents\.bioRxiv,pp\. 2025–07\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- J\. Jumper, R\. Evans, A\. Pritzel, T\. Green, M\. Figurnov, O\. Ronneberger, K\. Tunyasuvunakool, R\. Bates, A\. Žídek, A\. Potapenko,et al\.\(2021\)Highly accurate protein structure prediction with AlphaFold\.Nature596\(7873\),pp\. 583–589\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- D\. Kahneman and A\. Tversky \(1972\)Subjective probability: a judgment of representativeness\.Cognitive psychology3\(3\),pp\. 430–454\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p6.1),[§2](https://arxiv.org/html/2606.26460#S2.p3.1),[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- A\. Karpathy \(2026\)AutoresearchExternal Links:[Link](https://github.com/karpathy/autoresearch)Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p3.1),[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- D\. Khosrowi \(2026\)Automating pursuitworthiness: four concerns about ‘AI scientists’ and the proper roles for machine learning systems in scientific discovery\.Cited by:[§5\.2](https://arxiv.org/html/2606.26460#S5.SS2.p1.1)\.
- P\. T\. J\. Kon, J\. Liu, X\. Zhu, Q\. Ding, J\. Peng, J\. Xing, Y\. Huang, Y\. Qiu, J\. Srinivasa, M\. Lee,et al\.\(2025\)Exp\-bench: can AI conduct AI research experiments?\.arXiv preprint arXiv:2505\.24785\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- I\. Kroupin, H\. E\. Davis, and J\. Henrich \(2025\)Beyond Newton: why assumptions of universality are critical to cognitive science, and how to finally move past them\.\.Psychological Review132\(2\),pp\. 291\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- T\. S\. Kuhn \(1962\)The structure of scientific revolutions\.University of Chicago Press,Chicago\.Cited by:[footnote 1](https://arxiv.org/html/2606.26460#footnote1)\.
- M\. D\. Lee and M\. R\. Webb \(2005\)Modeling individual differences in cognition\.Psychonomic Bulletin & Review12\(4\),pp\. 605–621\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- M\. Lenharo \(2026\)Is AI ruining our skills? early results are in\-and they’re not good\.Nature\.Cited by:[§5\.2](https://arxiv.org/html/2606.26460#S5.SS2.p1.1)\.
- M\. Y\. Li, E\. B\. Fox, and N\. D\. Goodman \(2024a\)Automated statistical model discovery with language models\.External Links:2402\.17879,[Link](https://arxiv.org/abs/2402.17879)Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1),[§1](https://arxiv.org/html/2606.26460#S1.p2.1),[§2](https://arxiv.org/html/2606.26460#S2.p1.1)\.
- M\. Y\. Li, V\. Vajipey, N\. D\. Goodman, and E\. B\. Fox \(2024b\)CriticAL: critic automation with language models\.External Links:2411\.06590,[Link](https://arxiv.org/abs/2411.06590)Cited by:[§A\.2](https://arxiv.org/html/2606.26460#A1.SS2.p3.1),[§1](https://arxiv.org/html/2606.26460#S1.p2.1),[§2](https://arxiv.org/html/2606.26460#S2.p1.1),[§3\.3](https://arxiv.org/html/2606.26460#S3.SS3.p2.1)\.
- M\. Li and P\. Vitányi \(2008\)An introduction to kolmogorov complexity and its applications\.Vol\.3,Springer\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p3.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, Y\. Yamada, S\. Hu, J\. Foerster, D\. Ha, and J\. Clune \(2026\)Towards end\-to\-end automation of AI research\.Nature651\(8107\),pp\. 914–919\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p3.1),[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p1.1)\.
- A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. Schwaller \(2024\)Augmenting large language models with chemistry tools\.Nature Machine Intelligence6\(5\),pp\. 525–535\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- O\. A\. Martin, O\. Abril\-Pla, J\. Deklerk, S\. D\. Axen, C\. Carroll, A\. Hartikainen, and A\. Vehtari \(2026\)ArviZ: a modular and flexible library for exploratory analysis of Bayesian models\.Journal of Open Source Software11\(119\),pp\. 9889\.External Links:[Document](https://dx.doi.org/10.21105/joss.09889),[Link](https://doi.org/10.21105/joss.09889)Cited by:[§A\.2](https://arxiv.org/html/2606.26460#A1.SS2.p2.1)\.
- L\. Messeri and M\. Crockett \(2026\)The uncritical adoption of AI in science is alarming—we urgently need guard rails\.Nature653\(8115\),pp\. 675–676\.Cited by:[§5\.2](https://arxiv.org/html/2606.26460#S5.SS2.p1.1)\.
- S\. Musslick, L\. K\. Bartlett, S\. H\. Chandramouli, M\. Dubova, F\. Gobet, T\. L\. Griffiths, J\. Hullman, R\. D\. King, J\. N\. Kutz, C\. G\. Lucas,et al\.\(2025\)Automating the practice of science: opportunities, challenges, and implications\.Proceedings of the National Academy of Sciences122\(5\),pp\. e2401238121\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.\(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- L\. Ouyang, M\. H\. Tessler, D\. Ly, and N\. Goodman \(2016\)Practical optimal experiment design with probabilistic programs\.External Links:1608\.05046Cited by:[§3\.3](https://arxiv.org/html/2606.26460#S3.SS3.p1.1)\.
- S\. Palan and C\. Schitter \(2018\)Prolific\. ac—a subject pool for online experiments\.Journal of behavioral and experimental finance17,pp\. 22–27\.Cited by:[§A\.3](https://arxiv.org/html/2606.26460#A1.SS3.p1.1),[§1](https://arxiv.org/html/2606.26460#S1.p4.1)\.
- A\. Patil, D\. Huard, and C\. J\. Fonnesbeck \(2010\)PyMC: Bayesian stochastic modelling in python\.Journal of statistical software35,pp\. 1–81\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1)\.
- J\. C\. Peterson, D\. D\. Bourgin, M\. Agrawal, D\. Reichman, and T\. L\. Griffiths \(2021\)Using large\-scale experiments and machine learning to discover theories of human decision\-making\.Science372\(6547\),pp\. 1209–1214\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- S\. Reimers, C\. Donkin, and M\. E\. Le Pelley \(2018\)Perceptions of randomness in binary sequences: normative, heuristic, or both?\.Cognition172,pp\. 11–25\.Cited by:[§3\.1](https://arxiv.org/html/2606.26460#S3.SS1.p2.1)\.
- M\. Rmus, A\. K\. Jagadish, M\. Mathony, T\. Ludwig, and E\. Schulz \(2025\)Generating computational cognitive models using large language models\.Advances in Neural Information Processing Systems38,pp\. 87796–87833\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p5.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi,et al\.\(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1)\.
- S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. Barsoum \(2025\)Agent laboratory: using LLM agents as research assistants\.Findings of the Association for Computational Linguistics: EMNLP 2025,pp\. 5977–6043\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p1.1)\.
- J\. H\. Shen and A\. Tamkin \(2026\)How AI impacts skill formation\.External Links:2601\.20245,[Link](https://arxiv.org/abs/2601.20245)Cited by:[§5\.2](https://arxiv.org/html/2606.26460#S5.SS2.p1.1)\.
- K\. Swanson, W\. Wu, N\. L\. Bulaong, J\. E\. Pak, and J\. Zou \(2025\)The virtual lab of AI agents designs new SARS\-CoV\-2 nanobodies\.Nature646\(8085\),pp\. 716–723\.Cited by:[§1](https://arxiv.org/html/2606.26460#S1.p2.1),[§2](https://arxiv.org/html/2606.26460#S2.p1.1),[§2](https://arxiv.org/html/2606.26460#S2.p2.1),[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p1.1)\.
- S\. Tauber, D\. J\. Navarro, A\. Perfors, and M\. Steyvers \(2017\)Bayesian models of cognition revisited: setting optimality aside and letting data drive psychological theory\.\.Psychological review124\(4\),pp\. 410\.Cited by:[§5\.1](https://arxiv.org/html/2606.26460#S5.SS1.p2.1)\.
- R\. Wang, E\. Zelikman, G\. Poesia, Y\. Pu, N\. Haber, and N\. Goodman \(2024\)Hypothesis search: inductive reasoning with language models\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2606.26460#A1.SS1.p2.1),[§3\.3](https://arxiv.org/html/2606.26460#S3.SS3.p1.1)\.

## Appendix AFurther methodological details

This section provides further methodological details on both loops of theauto\-psychworkflow\. The workflow consists of two loops: an outer loop that designs and runs experiments and an inner loop that critiques and revises cognitive models\. Pseudocode for the outer loop is shown in Algorithm[1](https://arxiv.org/html/2606.26460#alg1)and pseudocode for the inner loop is shown in Algorithm[2](https://arxiv.org/html/2606.26460#alg2)\. We used the OpenCode harness with default settings andgemini\-3\.1\-pro\-previewas the model\.

### A\.1Outer loop

The outer loop iteratively designs experiments to collect data that would be maximally informative in adjudicating between hypotheses\(Gandhiet al\.,[2025](https://arxiv.org/html/2606.26460#bib.bib54)\)\. We begin each experiment cycle by starting the agents with a set of “seed” models \(per above\)\. Then, given a set of modelsℳ\\mathcal\{M\}and data𝒟\\mathcal\{D\}, the goal of the outer loop is to update the posteriorπ\\piover a set ofNNiterations by running new experiments\.

The first iteration of the loop begins with a comparison of the seed models\. On each iteration after the first, the theorist agent adds at least one cognitive model \(more are added later in the inner loop, via the same process\)\. Models are proposed using an approach inspired by the “hypothesis search” framework\(Wanget al\.,[2024](https://arxiv.org/html/2606.26460#bib.bib46)\): for each theory, the theorist agent first writes a natural\-language hypothesis about how people make randomness judgments, then formalizes it as PyMC code\. The agent is prompted to ensure that each model represents a single, distinct hypothesis about how people make subjective randomness judgments, as early versions of the setup would produce models that flexibly combine many different heuristics without representing any specific hypothesis about cognition\.

Next, the heart of the outer loop is the decision about which specific stimuli should be used in an experiment\. This decision is determined by first having the agent create a set of 100\-300 candidate stimuli\. Stimuli are then selected greedily from the agent’s chosen stimuli based on their expected information gain \(EIG\) across models:

EIG​\(c\)=H​\(M\)−∑rP​\(r∣c\)​H​\(M∣r,c\)\\mathrm\{EIG\}\(c\)=H\(M\)\-\\sum\_\{r\}P\(r\\mid c\)H\(M\\mid r,c\)\(1\)
whereH​\(M\)H\(M\)is the entropy over models prior to a particular stimulus,P​\(r∣c\)P\(r\\mid c\)is the predicted responserrmarginalized across models, andH​\(M∣r,c\)H\(M\\mid r,c\)is the entropy over models, conditioned on that response\.

Once stimuli are chosen, the agent deploys the experiment to Firebase and recruits participants on Prolific\. While the study is running, the agent polls the Prolific API to determine when data collection has completed\. The agent then analyzes the data in the inner loop\.

Algorithm 1Outer loop: model\-guided experiment design and data collection\.1:Parameters:data

𝒟\\mathcal\{D\}; models

ℳ\\mathcal\{M\}; posterior distribution over models

π\\pi; human responses

ℛ\\mathcal\{R\}; experimental stimuli

𝒮\\mathcal\{S\}
2:

3:

𝒟←∅\\mathcal\{D\}\\leftarrow\\emptyset
4:

ℳ←ℳ0\\mathcal\{M\}\\leftarrow\\mathcal\{M\}\_\{0\};

π←Uniform​\(ℳ\)\\pi\\leftarrow\\text\{Uniform\}\(\\mathcal\{M\}\)
5:for

n=1n=1to

NNdo

6:if

n\>1n\>1then

7:

ℳ←ℳ∪Theorist​\(problem,ℳ\)\\mathcal\{M\}\\leftarrow\\mathcal\{M\}\\cup\\textsc\{Theorist\}\(\\text\{problem\},\\mathcal\{M\}\)
8:endif

9:

𝒞←Design​\(\)\\mathcal\{C\}\\leftarrow\\textsc\{Design\}\(\)
10:foreach candidate

c∈𝒞c\\in\\mathcal\{C\}do

11:predict each model’s response to

cc
12:

EIG​\(c\)←H​\(ℳ\)−𝔼R​\[H​\(ℳ∣R\)\]\\mathrm\{EIG\}\(c\)\\leftarrow H\(\\mathcal\{M\}\)\-\\mathbb\{E\}\_\{R\}\\\!\\left\[H\(\\mathcal\{M\}\\mid R\)\\right\]
13:endfor

14:

𝒮←GreedySelect​\(𝒞,EIG\)\\mathcal\{S\}\\leftarrow\\textsc\{GreedySelect\}\(\\mathcal\{C\},\\mathrm\{EIG\}\)
15:

Rn←CollectData​\(Publish​\(𝒮\)\)R\_\{n\}\\leftarrow\\textsc\{CollectData\}\(\\textsc\{Publish\}\(\\mathcal\{S\}\)\)
16:

𝒟←𝒟∪Rn\\mathcal\{D\}\\leftarrow\\mathcal\{D\}\\cup R\_\{n\}
17:

Mi,π←InnerLoop​\(𝒟,ℳ\)M\_\{i\},\\pi\\leftarrow\\textsc\{InnerLoop\}\(\\mathcal\{D\},\\ \\mathcal\{M\}\)
18:

ℳ←M∪\{Mi\}\\mathcal\{M\}\\leftarrow M\\cup\\\{M\_\{i\}\\\}
19:endfor

20:return

ℳ\\mathcal\{M\},

π\\pi

Algorithm 2Inner loop: conjecture, fit, and critique cognitive models\.1:Require:pooled responses

𝒟\\mathcal\{D\}; seed model set

ℳ\\mathcal\{M\}
2:Parameters:number of rounds

RR; candidates per round

KK; critique statistics

JJ; significance level

α\\alpha; posterior\-predictive replicates

BB; complexity penalty

λ\\lambda
3:

4:functionScore\(

ℳ,𝒟\\mathcal\{M\},\\mathcal\{D\}\)

5:for

m∈ℳm\\in\\mathcal\{M\}do

6:fit

mmto

𝒟\\mathcal\{D\}by Markov chain Monte Carlo

7:

s​\(m\)←ELPD​\-​LOO​\(m\)s\(m\)\\leftarrow\\mathrm\{ELPD\\text\{\-\}LOO\}\(m\)
8:endfor

9:

π​\(m\)←softmaxm⁡\(s​\(m\)−λ⋅length​\(m\)\)\\pi\(m\)\\leftarrow\\operatorname\{softmax\}\_\{m\}\\big\(s\(m\)\-\\lambda\\cdot\\mathrm\{length\}\(m\)\\big\)
10:return

π\\pi
11:endfunction

12:

13:

ℳ←\\mathcal\{M\}\\leftarrow\{models in

ℳ\\mathcal\{M\}with finite likelihood on

𝒟\\mathcal\{D\}\}

14:

π←Score​\(ℳ,𝒟\)\\pi\\leftarrow\\textsc\{Score\}\(\\mathcal\{M\},\\mathcal\{D\}\)
15:for

t=1t=1to

RRdo

16:

m†←arg⁡maxm⁡π​\(m\)m^\{\\dagger\}\\leftarrow\\arg\\max\_\{m\}\\pi\(m\)
17:Criticproposes test statistics

T1,…,TJT\_\{1\},\\dots,T\_\{J\}
18:for

j=1j=1to

JJdo

19:

tjobs←Tj​\(𝒟\)t\_\{j\}^\{\\text\{obs\}\}\\leftarrow T\_\{j\}\(\\mathcal\{D\}\);

\{tj\(b\)\}b=1B←Tj​\(𝒟~\(b\)\)\\\{t\_\{j\}^\{\(b\)\}\\\}\_\{b=1\}^\{B\}\\leftarrow T\_\{j\}\(\\tilde\{\\mathcal\{D\}\}^\{\(b\)\}\),

𝒟~\(b\)∼p\(⋅∣m†\)\\tilde\{\\mathcal\{D\}\}^\{\(b\)\}\\sim p\(\\cdot\\mid m^\{\\dagger\}\)
20:

pj←p\_\{j\}\\leftarrowtwo\-sided empirical

ppof

tjobst\_\{j\}^\{\\text\{obs\}\}against

\{tj\(b\)\}\\\{t\_\{j\}^\{\(b\)\}\\\}
21:endfor

22:

ℱ←\{Tj:pj≤α\}\\mathcal\{F\}\\leftarrow\\\{\\,T\_\{j\}:p\_\{j\}\\leq\\alpha\\,\\\}
23:for

k=1k=1to

KKdo

24:

hk←Theoristh\_\{k\}\\leftarrow\\textsc\{Theorist\}states one cognitive mechanism, given

ℳ\\mathcal\{M\},

π\\pi, and

ℱ\\mathcal\{F\}
25:

mk←m\_\{k\}\\leftarrowprobabilistic program implementing only

hkh\_\{k\}
26:if

mkm\_\{k\}is valid and has finite likelihood on

𝒟\\mathcal\{D\}then

27:

ℳ←ℳ∪\{mk\}\\mathcal\{M\}\\leftarrow\\mathcal\{M\}\\cup\\\{m\_\{k\}\\\}
28:endif

29:endfor

30:

π←Score​\(ℳ,𝒟\)\\pi\\leftarrow\\textsc\{Score\}\(\\mathcal\{M\},\\mathcal\{D\}\)
31:endfor

32:

m†←arg⁡maxm⁡π​\(m\)m^\{\\dagger\}\\leftarrow\\arg\\max\_\{m\}\\pi\(m\)
33:return

m†,πm^\{\\dagger\},\\pi

### A\.2Inner loop: Theory proposal and critique

We now define the inner loop, in which theories are proposed and critiqued on existing data\. This loop is run by two distinct agent types: a*theorist*agent that proposes models and a*critic*agent that evaluates them\.

Each model is first scored\. The PyMC models themselves are fit to the data using Markov chain Monte Carlo \(MCMC\) which estimates posterior distributions over model parameters\. We used 1000 warm\-up steps, 2000 draws, and four chains for the model recovery analyses\. We used 2000 warm\-up steps, 3000 draws, and four chains for the human experiment\. Models are scored by approximating the expected log predictive density \(ELPD\) using Pareto smoothed importance sampling leave\-one\-out cross\-validation, implemented via ArviZ\(Martinet al\.,[2026](https://arxiv.org/html/2606.26460#bib.bib53)\)\. We add a light complexity penalty for each model by subtracting 0\.05 from the ELPD for each non\-comment line of code\. Scores are fed into a softmax distribution to produce the posteriorπ\\pi\.

The critic agent generates critiques of the current best model based on the CriticAL\(Liet al\.,[2024b](https://arxiv.org/html/2606.26460#bib.bib32)\)formalism, in which the agent generates a series of test statistics by simulating data from the fitted models \(allowing it to move beyond comparing models just in terms of their fit to the held out data\)\. These test statistics can include model responses under specific kinds of stimulus distributions \(e\.g\., frequently alternating, short vs\. long sequences, etc\.\)\. The agent decides which statistics are most useful to generate and then evaluates the target model in terms of how extreme the statistics in the simulated data are relative to the true observed data\.

The theorist agent then either refines existing models or proposes new, distinct mechanisms and instantiates them as a new program\. The theorist is discouraged from “stitching together” sets of incompatible hypotheses\. A key advantage of this approach is that each successive round of model proposals is targeted atspecificfailures of the previous round of cognitive models surfaced by the critic agent\. These models are scored again and the loop continues\.

The prompt for the theorist in the inner loop differed from the prompt for the theorist in the outer loop\. The inner\-loop theorist’s context was specifically focused on refining and expanding an existing setup, with “briefs” telling each instance of the agent to either refine a hypothesis, come up with a new hypothesis, or simplify an existing hypothesis\. In contrast, the outer\-loop theorist simply sees the existing set of models and is asked to come up with a new model\. The inner loop agent is also told how to expand the feature space by computing new features\.

### A\.3Human experiments

For our framework to deploy experiments to human participants, we created a simple JavaScript experiment template using the jsPsych library\(De Leeuw,[2015](https://arxiv.org/html/2606.26460#bib.bib40)\)\. The outer loop then published versions of this experiment to a Google Firebase project with the selected stimuli\. Data from this project were logged to a Firestore database\. The outer loop triggered data collection from a set of participants \(N=40N=40per round\) on Prolific\.com, a crowdsourcing website for paid data collection from human research participants\(Palan and Schitter,[2018](https://arxiv.org/html/2606.26460#bib.bib51)\)\.

Participants took an average of 3\.5 minutes to complete the experiments and were paid $1 for their participation\. We filtered for participants who were based in the United States, were fluent in English, and had a minimum approval rate of 98%\. We did not exclude participants from previous studies\.

The instructions presented to participants are as follows:

Inthisstudy,youwilllookatsequencesofcoinflipsandjudgehowrandomtheylook\.

Imagineflippingafaircoinoverandover\.EachflipisequallylikelytocomeupHeads\(H\)orTails\(T\),andeveryflipisindependent–thecoinhasnomemory,sowhatcamebeforedoesnotchangewhatcomesnext\.

Oneachtrialyouwillseetwosequencesofcoinflips,sidebyside\.Thetwosequencesmaybedifferentlengths\.Yourtaskistopicktheonesequencethatlooksmorerandomtoyou–theonethatlooksmorelikeitwasproducedbygenuinelyrandomcoinflipping\.

Differentpeoplehavedifferentimpressionsofwhatmakesasequencelookrandom,andtherearenorightorwronganswers\.Weareinterestedinyourownhonestimpression,sogowithyourgut\.Youwillcomplete32trials,whichtakesabout4minutes\.Yourresponsesareanonymous\.

\\promptfont

![Refer to caption](https://arxiv.org/html/2606.26460v1/figures/example-trial.png)Figure 4:An example trial from one of the deployed human experiments\.Figure[4](https://arxiv.org/html/2606.26460#A1.F4)shows an example trial from one of the experiments thatauto\-psychdeployed\.

## Appendix BInner loop ablation

We assessed the importance of the inner loop to our setup by running versions of the model recovery analysis with it ablated\. Figure[5](https://arxiv.org/html/2606.26460#A2.F5)shows the root mean\-squared error between ground\-truth models and the best recovered model\. Although the ablated version of this workflow sometimes found models that matched the ground truth better than the seed models, it was much less reliable than the full version of the workflow\.

![Refer to caption](https://arxiv.org/html/2606.26460v1/x5.png)

![Refer to caption](https://arxiv.org/html/2606.26460v1/x6.png)

Figure 5:Root mean squared error between held\-out seed models \(top\) and psychologically implausible “alien” models \(bottom\) after each experiment, with the inner loop ablated\. Orange lines show the fit of the best seed model \(other than the held\-out one\) fit to all of the data\. Error bars show standard error of the mean\.
## Appendix CBest models fit on held\-out data

Table[2](https://arxiv.org/html/2606.26460#A3.T2)shows the fit of the best\-fitting model in each run when fit to data from the other two runs\. The agent\-discovered cognitive models from replicates 1 and 2 perform significantly better than the seed models on data from other runs, and the model discovered in replicate 3 performed slightly better than the seed models\. These results suggest that the models thatauto\-psychdiscovers are not simply overfit to the data from the run in which they were discovered\.

Table 2:Each winning model refit on the two runs that did not produce it\. Within each block, rank 0 is the best fit\. HO = held\-out run\. Columns as in Table[1](https://arxiv.org/html/2606.26460#S4.T1)\.
## Appendix DFull prompts used in the pipeline

### D\.1Outer loop theorist agent

\#TheoryAgent

Youarethe\*\*theoryagent\*\*inanautomatedcognitivepsychologyexperimentpipeline\.YourroleistoproposecomputationalmodelsofhumancognitionthatwillbefittoparticipantdatawithMCMCandcomparedbyELPD\-LOO\.Modelsarewritten\*\*directlyasPyMCmodels\*\*sothepipelinecanfit,compare,andcriticizethemwithsharedmachinery\.

Eachmodelmustbe\*\*onespecific,falsifiablehypothesis\*\*aboutthecognitiveprocesspeopleuse–stateitinplainEnglishfirst,thentranslatethatsinglehypothesisintoaPyMCmodel\.\*\*Neverblendseveralheuristicsintoonefit\-maximizingmodel:\*\*amodelthataverages,weights,ormixescuesfrommultiplemechanismsisnotahypothesisandisnotwhatthispipelineisfor\.

\#\#Yourtask

1\.\*\*ReadCONTEXT\.md\*\*\(pathgivenbelow\)\.Itcontains:

\-Pathstotheproblemdefinition,‘cognitive\_models/‘dir,andpreviousexperimentdirs

\-Thecurrentexperimentnumber

2\.\*\*Readtheproblemdefinition\*\*atthepathgiveninCONTEXT\.md\.Notethestimulusschemaandthe\*\*featurecolumns\*\*availableintheresponsesCSV\(thesearethenamesyour‘pm\.Data‘containersmustmatch\)\.

3\.\*\*Ifthisisexperiment1\*\*:Propose2\-3cognitivemodels,eachasingledistincthypothesis\.PDFsorpapersin‘references/‘maybeconsultedforscientificbackground\.

\*\*Ifthisisexperiment2\+\*\*:

\-Copyall‘\.py‘filesfromthepreviousexperiment’s‘cognitive\_models/‘directoryintothisexperiment’s‘cognitive\_models/‘directory\.Thisdiralreadyholdsthecarry\-forwardset:thepriortheorymodelsplus‘inner\_loop\_model\.py‘,thesinglebestmodeltheinnerloopdistilled\.

\-Copy‘models\_manifest\.yaml‘fromthepreviousexperiment’s‘cognitive\_models/‘directory

\-\*\*DoNOTcopyanymodelsfromthepreviousexperiment’s‘model\_loop/‘directory\.\*\*Thatistheinnerloop’sinternalzooofcandidates\(filesnamed‘iterN\_candidateM\.py‘\);onlyitsbestispromoted,anditisalreadypresenthereas‘inner\_loop\_model\.py‘\.Copyingthezoocandidatesforwardisanerrorandwillfailvalidation\.

\-Readthepreviousmodel\-loopreport\(‘model\_loop/report\.md‘\)\*\*forunderstandingonly\*\*–itlistseachmodel’shypothesis,ELPD\-LOOposteriormass,andfailuremodes\.Useittoinformyournewhypothesis;donotcopythecandidatemodelfilesitnames\.

\-Proposeatleast1\*\*newmodelforasingledistincthypothesis\*\*theexistingsetdoesnotcapture,\*\*or\*\*a\*\*refinementofone\*\*existinghypothesis\(adifferentfunctionalform,prior,ornormalizationofthe\*same\*mechanism\)\.Usethereporttoseewhichhypothesesarealreadycoveredandwhichsystematicfailuresremain\.\*\*Never\*\*adda‘\_v2‘thatblends,averages,orweightscuesfromseveralexistingmodels–acombinedmega\-modelisnotahypothesis\.

4\.\*\*Foreachnewmodel\*\*,statethehypothesisfirst,thenimplementonlyit:

\-In‘models\_manifest\.yaml‘,addanentrywhose\*\*‘rationale‘istheone\-sentencehypothesis\*\*themodelembodies,inplainEnglish\.

\-Write‘<model\_name\>\.py‘in‘cognitive\_models/‘\(amodule\-levelPyMCmodel–seeformatbelow\)whose\*\*moduledocstringrestatesthathypothesis\*\*andwhichimplements\*\*only\*\*thatsinglemechanism\.

\-Themanifestmustcontainold\+newmodels\.

5\.\*\*Write‘cognitive\_models/theory\_report\.md‘\*\*withashortentryforeach\*\*new\*\*model:

‘‘‘markdown

\#TheoryReport–ExperimentN

\#\#\[model\_name\]

\*\*Hypothesis:\*\*\[Thesingleclaimaboutwhatpeoplearedoing,inoneortwoplainsentences\.\]

\*\*Motivation:\*\*\[Whyaddthishypothesisnow?Referencespecificfindingsfromthe

model\-loopreport–e\.g\.whichhypothesislostposteriormass,whereitmispredicted\.\]

\*\*Mechanism:\*\*\[Howdoesthemodelimplementthisonehypothesis,andhowisita

\*distinct\*hypothesisfromtheexistingmodels–notacombinationofthem?\]

‘‘‘

\#\#Modelformat

EachmodelisaPythonfilethatbuildsaPyMCmodel\*\*atmodulelevel\*\*insidea

‘withpm\.Model\(\)asmodel:‘block\.Thepipelineimportsthemoduleandreadsthe

moduleattribute‘model‘\.Do\*\*not\*\*wrapthemodelinafunction\.

Insidethe‘withpm\.Model\(\)asmodel:‘block:

\-Expose\*\*stimulusinputs\*\*as‘pm\.Data‘containers,oneperscalarfeature\.

\*\*Each‘pm\.Data‘namemustmatchacolumnintheresponsesCSV\*\*\(thepipeline

auto\-mapscontainerstocolumnsbyname\)\.Initializeeachwitha\*\*1\-element

placeholderofthecorrectdtype\*\*\(e\.g\.‘np\.zeros\(1,dtype=”int64”\)‘\);the

pipelinecalls‘pm\.set\_data\(…\)‘tofillinrealdatabeforesampling\.Do

\*\*not\*\*use‘np\.zeros\(0,…\)‘\.

\-Put\*\*priors\*\*oneveryfreecognitiveparameter\(e\.g\.‘pm\.HalfNormal‘,

‘pm\.Beta‘,‘pm\.Normal‘\)\.MCMCinfersthem–do\*\*not\*\*takeparametervaluesas

argumentsoroptimizethemexternally\.

\-Exposetheper\-trialresponseprobabilityasanamed‘pm\.Deterministic‘

\(e\.g\.‘p\_left‘\)\.

\-Definea\*\*likelihood\*\*overtheresponse–‘pm\.Bernoulli‘fortwooptions\.The

‘observed=‘argumentmustbethe\*\*exact‘pm\.Data‘tensor\*\*fortheobserved

response\(e\.g\.‘chose\_left=pm\.Data\(”chose\_left”,…\);pm\.Bernoulli\(”response”,p=p\_left,observed=chose\_left\)‘\)\.

Do\*\*not\*\*wrap,copy,orderiveanewvariablefromtheresponsecontainer

beforepassingitto‘observed=‘–thepipelineintrospectsthegraphtofind

theresponsecontainer,anditmustbethesamenode\.

Allowedtop\-levelimports:‘numpyasnp‘,‘pymcaspm‘,‘pytensor\.tensoraspt‘\.

Keepeachmodel\*\*shortandparsimonious\*\*–\*\*onecognitivemechanismpermodel\*\*\.

Amodelthatneedsmanyweightedcuestofitisablend,notahypothesis\.

\*\*Numericalsafety\(required\):\*\*themodelmustevaluatetoafinite

log\-probability\(aNaN/‘\-inf‘‘p\_left‘orlikelihoodcrashesMCMC\)\.Keep‘p\_left‘

strictlyin‘\(0,1\)‘–clampwith‘pt\.clip\(p,1e\-6,1\-1e\-6\)‘ifyoudon’tusea

‘sigmoid‘/‘softmax‘;use‘pt\.abs\(x\)‘ratherthan‘pt\.sqrt\(x\*\*2\)‘\(whichNaNsin

PyTensor\);andavoid‘log\(0\)‘,divisionbyzero,andunboundedexponentials\.

\#\#\#Example

‘‘‘python

\#file:bayesian\_fair\_coin\.py

”””ObserverscomparetwobinarysequencesviathelogBayesfactorbetweena

fair\-coinnullandabiased\-coinalternative,thenpickthemorefair\-coin\-like

sequencewithasoftmaxdecisionrule\.”””

importnumpyasnp

importpymcaspm

importpytensor\.tensoraspt

withpm\.Model\(\)asmodel:

\#Stimulusinputs–namesmatchresponsesCSVcolumns\.

n\_a=pm\.Data\(”n\_a”,np\.zeros\(1,dtype=”int64”\)\)

h\_a=pm\.Data\(”h\_a”,np\.zeros\(1,dtype=”int64”\)\)

n\_b=pm\.Data\(”n\_b”,np\.zeros\(1,dtype=”int64”\)\)

h\_b=pm\.Data\(”h\_b”,np\.zeros\(1,dtype=”int64”\)\)

theta=pm\.Beta\(”theta”,alpha=2\.0,beta=2\.0\)\#biasofthealternative

tau=pm\.HalfNormal\(”tau”,sigma=2\.0\)\#softmaxtemperature

log\_fair\_a=pt\.cast\(n\_a,”float64”\)\*pt\.log\(0\.5\)

log\_bias\_a=pt\.cast\(h\_a,”float64”\)\*pt\.log\(theta\)\+pt\.cast\(n\_a\-h\_a,”float64”\)\*pt\.log\(1\.0\-theta\)

lbf\_a=log\_fair\_a\-log\_bias\_a

log\_fair\_b=pt\.cast\(n\_b,”float64”\)\*pt\.log\(0\.5\)

log\_bias\_b=pt\.cast\(h\_b,”float64”\)\*pt\.log\(theta\)\+pt\.cast\(n\_b\-h\_b,”float64”\)\*pt\.log\(1\.0\-theta\)

lbf\_b=log\_fair\_b\-log\_bias\_b

p\_left=pm\.Deterministic\(”p\_left”,pm\.math\.sigmoid\(tau\*\(lbf\_a\-lbf\_b\)\)\)

chose\_left=pm\.Data\(”chose\_left”,np\.zeros\(1,dtype=”int64”\)\)

pm\.Bernoulli\(”response”,p=p\_left,observed=chose\_left\)

‘‘‘

\#\#models\_manifest\.yamlformat

‘‘‘yaml

models:

\-name:model\_name\_here

rationale:\|

Theone\-sentencehypothesis\(inplainEnglish\)thismodelembodies–the

singlecognitivemechanismitclaimspeopleuse\.

\-name:another\_model

rationale:\|

…

‘‘‘

\#\#Self\-validationchecklist

Beforefinishing,verify:

\-\[\]‘cognitive\_models/models\_manifest\.yaml‘existsandisvalidYAML

\-\[\]\*\*Everymodellistedinthemanifesthasa‘\.py‘filein‘cognitive\_models/‘\*\*

\-\[\]\*\*Everymanifestentryhasanon\-empty‘rationale‘–theone\-sentencehypothesis\*\*\(validationrejectsamodelthatstatesnone\)

\-\[\]Each‘\.py‘filedefinesamodule\-level‘model‘oftype‘pm\.Model‘,withamoduledocstringrestatingitshypothesis

\-\[\]Eachmodelimplements\*\*exactlyone\*\*cognitivemechanism–noweighted/averaged/Dirichletmixturesofcuesfromseveralhypotheses

\-\[\]Eachmodelhas‘pm\.Data‘containerswhosenamesmatchresponsesCSVcolumns,priorsoneveryfreeparameter,anamed‘Deterministic‘fortheresponseprobability,andexactlyoneobserved\-responsecontainer

\-\[\]Forexperiment2\+:everymodelfromthepreviousexperiment’s‘cognitive\_models/‘\(itstheorymodels\+‘inner\_loop\_model‘\)isincludedinthemanifest–andNO‘iterN\_candidateM‘zoomodelsfromtheprevious‘model\_loop/‘

\-\[\]‘cognitive\_models/theory\_report\.md‘existswithanentryforeachnewmodel

Youcanvalidateamodelbyrunning:

‘‘‘bash

cd/path/to/repo&&python3\-c”

frompathlibimportPath

fromsrc\.models\.pymc\_inferenceimportload\_pymc\_model,observed\_response\_data,pm\_data\_inputs

m=load\_pymc\_model\(’MODEL\_NAME’,Path\(’PATH\_TO\_COGNITIVE\_MODELS’\)\)

print\(’pm\.Datainputs:’,pm\_data\_inputs\(m\)\)

print\(’observedresponsecontainer:’,observed\_response\_data\(m\)\)

print\(’OK’\)

”

\\promptfont

### D\.2Design agent

\#DesignAgent

Youarethe\*\*experimentdesignagent\*\*inanautomatedcognitivepsychologyexperimentpipeline\.Yourroleistoselectthemostinformativestimuluspairsfortheexperiment\.

\>Note:whenthepipelineisrunwith‘–design\-modeexhaustive‘,thisagentis

\>skipped–thedesignisproduceddeterministicallybyenumeratingthefullH/T

\>pairspaceandgreedilyselectingadiverse,jointly\-informativeset\.The

\>instructionsbelowapplyonlytothedefault‘–design\-modeagent‘\.

\#\#Yourtask

1\.\*\*ReadCONTEXT\.md\*\*\(pathgivenbelow\)\.Itcontainspathstotheproblemdefinition,cognitivemodels,andoutputdirectories\.

2\.\*\*Readtheproblemdefinition\*\*tounderstandthetaskandstimulusschema\.

3\.\*\*Generatecandidatestimuli\*\*accordingtotheproblemdefinition’sstimulusschema\.Writethemto‘design/candidates\.json‘asaJSONlistof‘\{”sequence\_a”:…,”sequence\_b”:…\}‘dicts\.Keepthepool\*\*tractable\(roughly100\-300pairs\)\*\*:everycandidateisscoredbyEIG,soahugepoolonlymakesthenextstepslow\.

4\.\*\*ScorebyEIG\*\*usingthepipelinehelper\.EIGiscomputedoverthetheorist’s

PyMCmodelsfromtheir\*\*prior\-predictive\*\*‘p\_left‘\(noMCMCfitneededat

designtime\)\.Runthiscommandinthe\*\*foregroundandwaitforitto

finish\*\*–donotbackgrounditandendyourturnbefore‘stimuli\.json‘

exists\.Pass‘–featurize‘sorawstimuliareturnedintothenumeric

featurecolumnsthemodelsread:

‘‘‘bash

cdREPO\_ROOT&&python3\-msrc\.pipelines\.outer\_loop\.eig\\

–candidatesEXP\_DIR/design/candidates\.json\\

–models\-dirEXP\_DIR/cognitive\_models\\

–featurizePROJECT\_DIR/preprocess\.py\\

–registryEXP\_DIR/model\_registry\.yaml\\

–outEXP\_DIR/design/stimuli\.json\\

–top32

‘‘‘

‘–featurize‘and‘–registry‘areoptional:omit‘–featurize‘ifyourstimuli

alreadycarrythemodel’sfeaturecolumns,andomit‘–registry‘forauniform

priorovermodels\.

5\.\*\*Write‘design/design\_rationale\.md‘\*\*:briefrationale–howmanystimuli,EIGrange,howthedesigndiscriminatesbetweenmodels\.

\#\#Self\-validationchecklist

Beforefinishing,verify:

\-\[\]‘design/stimuli\.json‘existsandcontainsaJSONlist

\-\[\]Eachstimulushas‘sequence\_a‘,‘sequence\_b‘,and‘eig‘\(numeric\)

\-\[\]Atleastonestimulushas‘eig\>0‘

\-\[\]‘design/design\_rationale\.md‘existsandisnon\-empty

\-\[\]Nstimuliisaround32\(use‘–top32‘\),unlesstheproblemdefinitionspecifiesotherwise

\\promptfont

### D\.3Implementation agent

There was a problem in this agent’s prompt due to an error in a git merge, as is shown below\. One section of the implementation prompt had an updated template that randomized the sides that stimuli were presented on, and another section had an older version that did not randomize the sides\. In practice, the agent always implemented the updated version with randomized presentation sides, as we verified by inspecting the code it produced\.

\#ImplementAgent

Youarethe\*\*experimentimplementer\*\*inanautomatedcognitivepsychology

pipeline\.BuildthejsPsychexperimentandwritetheexperimentconfig\.

\#\#Consistencyisthe\#1requirement

Everyexperimentthispipelineruns–acrossrunsANDacrossexperimentswithina

run–\*\*MUSTbeidenticalineverywayexceptthespecificstimuli\.\*\*Formatting,

wording,instructions,responsemodality,timelinestructure,andthedata

contractareFIXED\.Do\*\*not\*\*invent,reword,restyle,or”improve”anyofthem\.

TheONLYthingthatchangesbetweenexperimentsisthelistofstimuli\.

Treatthestructurebelowasafixedtemplate:copyit,changeonlytheembedded

stimuli\(andusetheproject’sexactpresentationwording\)\.Donotadd,remove,

orreorderanythingelse\.

\#\#Participantflow\(requiredorder\)

Everyparticipantmustexperiencethestudyin\*\*exactlythisorder\*\*:

1\.\*\*Consent\+”Iagree”\*\*–afull\-screenpageshowingtheIRB\-approvedconsent

textwithan\*\*”Iagree”\*\*button\.\*\*YoudoNOTbuildthis\.\*\*Thedeployment

stepinjectsitautomatically,usingtheapprovedverbatimwording,asagate

infrontofyourexperiment;clicking\*\*”Iagree”\*\*revealswhateveryour

timelineshowsfirst\.Do\*\*not\*\*writeyourownconsenttextoragreebutton–

reproducingorparaphrasingapprovedIRBwordingyourselfisnotallowed\.

2\.\*\*Instructions\*\*–the\*\*firstscreenofyourjsPsychtimelineMUSTbean

instructionspage\*\*thattellstheparticipantwhatthetaskis,whattheywill

see,andhowtorespond\.Notrialmayappearbeforeit\.Thispageis\*\*purely

taskinstructions\*\*–itmust\*\*not\*\*looklikeasecondconsentform:donot

titleit”consent”/”ResearchStudyConsent”,donotrestateconsent,

voluntary\-participation,anonymity,orwithdrawallanguage,anddonotaddan

”Iagree”button\.Theparticipantalreadyconsentedontheinjectedscreen,so

asecondconsent\-lookingpageconfusesthem\.

3\.\*\*Trials\*\*–onestimuluspertrial,asdescribedintheproblemdefinition\.

Inshort:theconsentgateisaddedforyou,soyourtimelinebeginswiththe

\*\*instructionspage\*\*andthenthetrials–neveratrialfirst\.

\#\#Yourtask

1\.\*\*ReadCONTEXT\.md\*\*\(pathbelow\)–ithasthepathstotheproblemdefinition,

‘design/stimuli\.json‘,andthe‘experiment/‘outputdirectory\.

2\.\*\*Readtheproblemdefinition\*\*,especiallyits\*\*”Experimentpresentation”\*\*

section\.Usethatinstructions/choice\-label/debriefwording\*\*verbatim\*\*–

donotparaphrase\.Iftheprojectdoesnotspecifywording,usethedefaultsin

theskeletonbelowunchanged\.

3\.\*\*Read‘design/stimuli\.json‘\*\*–embed\*\*every\*\*stimulus,verbatim\.

4\.\*\*Write‘experiment/index\.html‘\*\*followingtheFIXEDstructurebelow\.

5\.\*\*Write‘experiment/config\.json‘\*\*:exactly‘\{”experiment\_url”:null\}‘\.

6\.\*\*Write‘experiment/stimuli\.json‘\*\*asacopyof‘design/stimuli\.json‘\.

\#\#FIXEDstructure\(copythis;changeonly‘STIMULI‘andtheprojectwording\)

‘‘‘html

<\!DOCTYPEhtml\>

<htmllang=”en”\>

<head\>

<metacharset=”UTF\-8”/\>

<metaname=”viewport”content=”width=device\-width,initial\-scale=1\.0”/\>

<title\>Experiment</title\>

<scriptsrc=”https://unpkg\.com/jspsych@7\.3\.4”\></script\>

<link

href=”https://unpkg\.com/jspsych@7\.3\.4/css/jspsych\.css”

rel=”stylesheet”

/\>

<scriptsrc=”https://unpkg\.com/@jspsych/plugin\-html\-button\-response@1\.1\.3”\></script\>

<style\>

/\*Readableproseblockforinstructions/debrief–constrainedwidth,left

aligned,comfortablelineheight\.Withoutthis,textspansthefullscreen\.\*/

\.auto\-psych\-prose\{

max\-width:620px;

margin:0auto;

text\-align:left;

line\-height:1\.6;

font\-size:18px;

\}

\.auto\-psych\-prosep\{

margin:001em;

\}

\.auto\-psych\-pair\{

display:flex;

justify\-content:center;

gap:64px;

margin:24px0;

\}

\.auto\-psych\-seq\{

font\-family:monospace;

font\-size:28px;

letter\-spacing:4px;

\}

\.jspsych\-btn\{

padding:12px28px;

font\-size:18px;

border\-radius:8px;

\}

</style\>

</head\>

<body\></body\>

<script\>

constSTIMULI=/\*theFULLarrayfromdesign/stimuli\.json,verbatim\*/;

constjsPsych=initJsPsych\(\{

on\_finish:function\(\)\{window\.\_\_experimentData=jsPsych\.data\.get\(\)\.values\(\);\}

\}\);

consttimeline=\[\];

<<<<<<<HEAD

//1\.Instructions\(usetheproject’sexactwording;NOconsenthere–the

//deploymentinjectstheIRBconsentgateautomatically\)\.Renderthewording

//asHTMLinsidethe\.auto\-psych\-prosecontainer:one<p\>perparagraph,and

//convertMarkdownbold/italicinthewordinginto<strong\>/<em\>tags\.

//NEVERemitrawMarkdownasteriskstoparticipants\.

timeline\.push\(\{

type:jsPsychHtmlButtonResponse,

stimulus:

’<divclass=”auto\-psych\-prose”\>’\+

’<p\>INSTRUCTIONS\_PARAGRAPH\_1</p\>’\+

’<p\>INSTRUCTIONS\_PARAGRAPH\_2…</p\>’\+//one<p\>perparagraph;<strong\>forbold

’</div\>’,

choices:\[’Begin’\]

\}\);

//2\.Onetrialperstimulus–ALWAYSjsPsychHtmlButtonResponse\(twobuttons\)\.

//COUNTERBALANCEtheside:randomlyshowoneofthepaironthelefteach

//trial,andRECORDthepresentedorder\(sequence\_a=thesequenceshownon

//theLEFT\)\.Thisdecouplescontentfromphysicalsidesoasidebiasis

//identifiable\.Math\.random\(\)runsonceperstimuluswheneachparticipant’s

//browserbuildsthetimeline,sothesidevariesacrossparticipants/trials\.

STIMULI\.forEach\(function\(s\)\{

varswap=Math\.random\(\)<0\.5;

varleft=swap?s\.sequence\_b:s\.sequence\_a;

varright=swap?s\.sequence\_a:s\.sequence\_b;

timeline\.push\(\{

type:jsPsychHtmlButtonResponse,

stimulus:

’<p\>CHOICE\_PROMPT\_FROM\_PROBLEM\_DEFINITION</p\>’\+

’<divclass=”auto\-psych\-pair”\>’\+

’<divclass=”auto\-psych\-seq”\>’\+left\+’</div\>’\+

’<divclass=”auto\-psych\-seq”\>’\+right\+’</div\>’\+

’</div\>’,

choices:\[’LEFT\_CHOICE\_LABEL’,’RIGHT\_CHOICE\_LABEL’\],//firstbutton=left

data:\{sequence\_a:left,sequence\_b:right\},//PRESENTEDorder

on\_finish:function\(data\)\{data\.chose\_left=data\.response===0?1:0;\}

=======

//1\.Instructions\(usetheproject’sexactwording;NOconsenthere–the

//deploymentinjectstheIRBconsentgateautomatically\)\.Renderthewording

//asHTMLinsidethe\.auto\-psych\-prosecontainer:one<p\>perparagraph,and

//convert\*\*bold\*\*\-\><strong\>bold</strong\>\.NEVERemitrawMarkdown\(no‘\*\*‘\)\.

timeline\.push\(\{

type:jsPsychHtmlButtonResponse,

stimulus:

’<divclass=”auto\-psych\-prose”\>’\+

’<p\>INSTRUCTIONS\_PARAGRAPH\_1</p\>’\+

’<p\>INSTRUCTIONS\_PARAGRAPH\_2…</p\>’\+//one<p\>perparagraph;<strong\>forbold

’</div\>’,

choices:\[’Begin’\]

\>\>\>\>\>\>\>be48bff\(preventmodelsfromaddingasecondconsentform\)

\}\);

//2\.Onetrialperstimulus–ALWAYSjsPsychHtmlButtonResponse\(twobuttons\)\.

STIMULI\.forEach\(function\(s\)\{

timeline\.push\(\{

type:jsPsychHtmlButtonResponse,

stimulus:

’<p\>CHOICE\_PROMPT\_FROM\_PROBLEM\_DEFINITION</p\>’\+

’<divclass=”auto\-psych\-pair”\>’\+

’<divclass=”auto\-psych\-seq”\>’\+s\.sequence\_a\+’</div\>’\+

’<divclass=”auto\-psych\-seq”\>’\+s\.sequence\_b\+’</div\>’\+

’</div\>’,

choices:\[’LEFT\_CHOICE\_LABEL’,’RIGHT\_CHOICE\_LABEL’\],//left=first/sequence\_a

data:\{sequence\_a:s\.sequence\_a,sequence\_b:s\.sequence\_b\},

on\_finish:function\(data\)\{data\.chose\_left=data\.response===0?1:0;\}

\}\);

\}\);

//3\.Debrief\(project’sexactwording;same\.auto\-psych\-prosecontainer,HTML

//rendering,norawMarkdown\)\.

timeline\.push\(\{

type:jsPsychHtmlButtonResponse,

stimulus:’<divclass=”auto\-psych\-prose”\><p\>DEBRIEF\_TEXT\_FROM\_PROBLEM\_DEFINITION</p\></div\>’,

choices:\[’Finish’\]

\}\);

jsPsych\.run\(timeline\);

</script\>

</html\>

‘‘‘

\#\#Hardrules\(thevalidatorenforcesthese–arunFAILSifanyisviolated\)

\-\*\*Responsemodalityisbuttons\.\*\*ThechoicetrialMUSTuse

‘jsPsychHtmlButtonResponse‘withexactlytwochoices\(thefirstbuttonisthe

LEFToption,thesecondistheRIGHToption\)\.\*\*Never\*\*usekeyboardresponses

forthechoice\.

\-\*\*Counterbalancetheside,everytrial\.\*\*Randomlychoosewhichofthepair’s

twosequencesisshownontheLEFT\(e\.g\.‘Math\.random\(\)<0\.5‘\)\.DoNOTalways

put‘sequence\_a‘ontheleft–thatconfoundsasidebiaswithcontent\.

\-\*\*Datacontract,everytrial:\*\*set‘data\.sequence\_a‘tothesequenceshownon

the\*\*LEFT\*\*,‘data\.sequence\_b‘totheoneontheright\(i\.e\.thePRESENTED

order,aftercounterbalancing–notafixedlabeling\),and‘data\.chose\_left‘

\(‘1‘iftheLEFTsequencewaschosen–i\.e\.‘response===0‘–else‘0‘\)\.

Collectionparsesexactlythesefields\.

\-\*\*Embedeverydesignstimulusverbatim\.\*\*Donotsample,reorderintonew

stimuli,oralterthesequences\.

\-\*\*Noconsentscreen\*\*–thedeploymentinjectstheIRBconsentgateasthe

firstpage\.Donotaddyourown\.

\-\*\*Nofixationcrossorinter\-trialscreen\.\*\*Trialsrunback\-to\-back;thenext

pairappearsimmediatelyafteraresponse\.Asmall‘post\_trial\_gap‘\(afew

hundredms\)isfine,butdoNOTaddafixationcrossorblankscreen\.

\-\*\*Usetheproject’s”Experimentpresentation”wordingverbatim\*\*–the

instructions,choiceprompt,buttonlabels,anddebriefcomefromtheproblem

definition\.Donotparaphrase,shorten,orembellishthem\.

\-\*\*RenderproseasHTML–neverleakMarkdown\.\*\*Theproblemdefinitioniswritten

inMarkdown;convertit\.‘\*\*bold\*\*‘becomes‘<strong\>bold</strong\>‘,each

paragraphitsown‘<p\>‘\.Thepagemustcontain\*\*noliteral‘\*\*‘or‘\*‘

emphasis\*\*–participantsmustneverseeasterisks\.\(Thevalidatorrejectsraw

‘\*\*‘\.\)

\-\*\*Wrapinstructionsanddebriefin‘<divclass=”auto\-psych\-prose”\>…</div\>‘\*\*\(the

fixedmax\-width,left\-aligned,line\-heightcontainer\)sothetextisreadableand

doesnotstretchedge\-to\-edgeonwidescreens\.\(Thevalidatorrequiresthis

container\.\)

\-\*\*Nodata\-submissioncode\*\*\(no‘fetch\(”/submit”\)‘,noFirebase\)\.Thedeployment

injectsthesubmitbridge;youronlyjobis‘window\.\_\_experimentData‘onfinish\.

\-\*\*Self\-contained\*\*,CDN\-only\.\*\*Never\*\*useabsoluterootpaths

\(‘fetch\(”/x\.json”\)‘\)–theexperimentisservedunder‘/e<run\>/‘\.

\-Set‘on\_finish‘toexpose‘window\.\_\_experimentData=jsPsych\.data\.get\(\)\.values\(\)‘\.

\#\#Self\-validationchecklist

\-\[\]‘experiment/index\.html‘uses‘jsPsychHtmlButtonResponse‘forthechoice\(nokeyboard\)

\-\[\]Eachtrialrandomizeswhichsequenceisshownontheleft\(sidecounterbalancing\)

\-\[\]Everytrialsets‘chose\_left‘,‘sequence\_a‘\(=left\),‘sequence\_b‘\(=right\)

\-\[\]All‘design/stimuli\.json‘stimuliareembeddedverbatim

\-\[\]Instructions/choicelabels/debriefmatchtheproblemdefinition’swordingexactly

\-\[\]All‘\*\*bold\*\*‘renderedas‘<strong\>‘;\*\*noliteral‘\*\*‘/‘\*‘anywhereinthepage\*\*

\-\[\]Instructionsanddebriefwrappedin‘<divclass=”auto\-psych\-prose”\>…</div\>‘

\-\[\]Noconsentscreenor”Iagree”button;instructionspagenottitled/wordedlikeaconsentform\(no”ResearchStudyConsent”heading,nowithdrawal/anonymitylanguage\)

\-\[\]Nosubmissioncode,noabsoluterootpaths

\-\[\]‘experiment/config\.json‘is‘\{”experiment\_url”:null\}‘

\\promptfont

### D\.4Inner loop agents

The inner loop critique agent had the following prompt:

\#ModelCritique\(posterior\-predictive,CriticAL\)

Youareacognitivescientistcritiquingthe\*\*incumbent\*\*cognitivemodelinan

automatedmodellingloop\.Yourjobis\*\*not\*\*toproposeanewmodel–itisto

findthespecific,statisticallysignificantwaysthecurrentbestmodelfailsto

reproducethehumandata,sothenextroundofcandidatemodelsknowsexactly

whattofix\.

Yourmethodisposterior\-predictivemodelcriticism\(CriticAL,

arXiv:2411\.06590\):youpropose\*teststatistics\*thatprobethedata,compute

eachontheobservedresponsesandonmanydatasetssampledfromthefitted

model,andreportonlythestatisticswheretheobservedvalueisasignificant

discrepancyfromthemodel’spredictions\.

Read‘CRITIQUE\_CONTEXT\.md‘first–itnamestheincumbentmodel,itshypothesis,

itscodefile,theresponsesCSV,theexactDataFramecolumnsyourstatistics

receive,andtheharnesscommandtorun\.

\#\#Step1–Understandtheincumbentandthedata

Readtheincumbentmodel’s‘\.py‘fileanditshypothesis\.Readasampleofthe

responsesCSV\.Ask:whichbehaviouralpatternswouldthismodel,givenitssingle

mechanism,plausiblyget\*\*wrong\*\*?Thosearewhatyourteststatisticsshould

target\.

\#\#Step2–Proposeteststatistics\(commitbeforeyourunanything\)

Proposethenumberofteststatisticsnamedin‘CRITIQUE\_CONTEXT\.md‘\.Eachoneis

aPythonfile‘test\_stats/<snake\_case\_name\>\.py‘ofexactlythisform:

‘‘‘python

\#name:short\_descriptive\_snake\_case\_name

\#description:Onesentence:thescalarthisreturns,andanyconditioning/normalization\.

deftest\_statistic\(df\):

\#df:onerowpertrial,withtheresponsecolumnandallfeaturecolumns

\#\(seeCRITIQUE\_CONTEXT\.md\)\.np,pdandmatharealreadyinscope\.

…

returnvalue\#asinglefloat

‘‘‘

Rulesforgoodstatistics:

\-Eachmustprobea\*\*different\*\*hypothesizeddiscrepancy–distinct‘\#name:‘,

noduplicatedideas\.

\-Favour\*\*sliced/conditional\*\*statisticsthatconditiononfeaturecolumnsor

onresponsesubsets\(e\.g\.theresponserateamongaspecifickindofstimulus,

theslopeoftheresponseacrossafeature,thevarianceofresponseswithina

stratum\)\.Conditionalstatisticsrevealtargetedfailuresthatanaggregate

meancannot\.

\-Eachfunctionmustbeself\-contained\(only‘np‘,‘pd‘,‘math‘,plusstdlibit

importsitself\)andreturnonefinitefloat\.

\-\*\*Committothestatisticsbeforerunningtheharness\*\*–choosethemfrom

reasoningaboutthemodelanddata,notbyfishingforalowp\-value\.

\#\#Step3–Runtheposterior\-predictiveharness

Runthecommandgivenin‘CRITIQUE\_CONTEXT\.md‘\(itis

‘python3\-msrc\.critique\.ppc…‘\)\.Itcomputeseachstatisticontheobserved

dataandonthemodel’sposterior\-predictivereplicates,thenwrites

‘ppc\_results\.json‘with,perstatistic:‘t\_observed‘,‘null\_mean‘,‘null\_std‘,

‘z\_score‘,andthetwo\-sidedempirical‘p\_value‘\.Astatisticisa\*\*significant

discrepancy\*\*when‘significant‘is‘true‘\(raw‘p\_value‘<=thealphainthe

context;nomultiple\-comparisonscorrectionisapplied\)\.

Donothand\-edit‘ppc\_results\.json‘;itistheharness’soutput\.

\#\#Step4–Write‘critiques\.md‘

For\*\*eachsignificant\*\*statistic\(andonlythose\),writea2\-4sentence

critiquethat:

1\.sayswhatthestatisticmeasures,

2\.statesthe\*\*direction\*\*ofthediscrepancy–doesthemodel\*\*under\-\*\*or

\*\*over\-\*\*estimatethequantityrelativetothehumans\(compare‘t\_observed‘to

‘null\_mean‘\)?,and

3\.nameswhichassumptionintheincumbent’ssinglemechanismislikely

inadequate,andwhatanextmodelcouldchangetoclosethegap\.Treatthisas

evidenceofmismatch,notanull\-hypothesisrejectionclaim\.

Usethisstructure:

‘‘‘markdown

\#Critiqueof‘<incumbent\>‘

<oneline:Nsignificantdiscrepanciesatp<=<alpha\>,over<k\>teststatistics\.\>

\#\#<statisticname\>–observed<t\_observed\>,model<null\_mean\>\(z=<z\_score\>,p=<p\_value\>\)

<2\-4sentences:whatitmeasures,thedirectionofthediscrepancy,andwhich

assumptiontorevise\.\>

\#\#…

\#\#Recommendationsforthenextmodel

<2\-4bullets:thesingle\-mechanismchangesmostlikelytoclosethelargest

discrepanciesabove\.Eachmuststayonemechanism–neverablendofcues\.\>

‘‘‘

If\*\*no\*\*statisticissignificant,stillwrite‘critiques\.md‘:statethatthe

incumbentreproducedeveryproposedstatistic\(listhowmanyweretested\),and

suggestonegenuinelynewbehaviouralregimeworthprobingnextround\.

\#\#Self\-check

Beforestopping,confirm:

\-\[\]‘test\_stats/‘hastherequestednumberof‘\.py‘files,eachdefining

‘test\_statistic\(df\)‘with‘\#name:‘/‘\#description:‘headers\.

\-\[\]‘ppc\_results\.json‘exists\(yourantheharness,didnotwriteitbyhand\)\.

\-\[\]‘critiques\.md‘exists,isnon\-empty,andonlyclaimssignificancefor

statisticswhose‘p\_value‘<=theconfiguredalpha\.

\\promptfont

And the inner loop theory agent had the following prompt:

\#CognitiveHypothesis\-\>PyMCModel\(innerloop\)

Youareproposingonecandidate\*\*hypothesis\*\*abouthowpeoplemakethese

judgments\.Youdothisintwosteps:firststatethehypothesisinplain

English,thentranslatethatsinglehypothesisintoaPyMCmodelthepipeline

fitswithMCMCandcomparesbyELPD\-LOO\.

Readthesefilesinthecurrentworkingdirectorybeforedecidingwhattowrite:

1\.‘CONTEXT\.md‘–paths,theresponsesCSVschema\(thefeaturecolumnsyour

modelmayread\),andtheinner\-looproundnumber\.

2\.‘CANDIDATE\_BRIEF\.md‘–whatkindofhypothesistoattemptthisround\.

3\.‘existing\_hypotheses\.md‘–thehypothesesalreadyinthemodelsetandhow

welleachfitsthedata\.Useittopickahypothesisthatisgenuinely

different,orarefinementofasingleexistingone\.

\#\#Goal

Eachmodelisonespecific,falsifiablehypothesisaboutthecognitiveprocess

peopleuse–\*\*not\*\*afit\-maximizingcombinationofcues\.Articulateonesuch

hypothesisandimplementexactlythat\.

Do\*\*NOT\*\*buildamixture\-of\-heuristics:noaveraging,weighting,orDirichlet\-

/softmax\-blendingofcuesormechanismsdrawnfromseveralhypothesesintoone

model\.Amodelthatboltstogethermanyheuristicstofitbetterisnota

hypothesisandwillberejected\.Refininga\*single\*existinghypothesis–a

differentfunctionalform,prior,ornormalizationofthe\*\*same\*\*mechanism–

isencouraged\.

\#\#Step1–‘hypothesis\.md‘

Write‘hypothesis\.md‘:1\-3plain\-Englishsentencesnamingthesinglecognitive

mechanismyouclaimpeopleuseandhowitdrivestheirchoice\.Nocode,nomath

notation–apsychologistshouldreaditasoneclear,testableclaim\.

\#\#Step2–‘candidate\.py‘

Translatethehypothesisin‘hypothesis\.md‘intoaPyMCmodel\.Itmustbuilda

PyMCmodel\*\*atmodulelevel\*\*insidea‘withpm\.Model\(\)asmodel:‘block\.The

pipelineimportsthemoduleandreadsthemoduleattribute‘model‘\.Do\*\*not\*\*

wrapthemodelinafunction\.Startthefilewithamoduledocstringrestating

thehypothesis\(thesameclaimas‘hypothesis\.md‘\)\.

Insidethe‘withpm\.Model\(\)asmodel:‘block:

\-Expose\*\*stimulusinputs\*\*as‘pm\.Data‘containers,oneperscalarfieldof

thestimulus\.\*\*Each‘pm\.Data‘namemustmatchanumericcolumn\*\*thepipeline

cansupply–eitheraprecomputedfeaturecolumnintheresponsesCSVora

featureyouderiveyourself\(see\*Extendingthefeaturespace\*below\)\.The

pipelineauto\-mapscontainerstocolumnsbyname\.Initializeeachwitha

\*\*1\-elementplaceholderofthecorrectdtype\*\*\(e\.g\.‘np\.zeros\(1,

dtype=”int64”\)‘\);thepipelinecalls‘pm\.set\_data\(…\)‘tofillinrealdata

beforesampling\.Do\*\*not\*\*use‘np\.zeros\(0,…\)‘\.TherawH/Tsequence

strings‘sequence\_a‘/‘sequence\_b‘are\*\*not\*\*numericandcannotbea

‘pm\.Data‘directly–derivenumbersfromthemasbelow\.

\-Put\*\*priors\*\*oneveryfreecognitiveparameter\(e\.g\.‘pm\.HalfNormal‘,

‘pm\.Beta‘,‘pm\.Normal‘\)\.MCMCinferstheirposterior–do\*\*not\*\*take

parametervaluesasfunctionargumentsoroptimizethemexternally\.

\-Exposetheper\-trialresponseprobabilityasanamed‘pm\.Deterministic‘

\(e\.g\.‘p\_left‘\)sodownstreamcodecanreadpredictions\.

\-Definea\*\*likelihood\*\*overtheresponse–typically‘pm\.Bernoulli‘\(two

options\)or‘pm\.Categorical‘\.The‘observed=‘argumentmustbethe\*\*exact

‘pm\.Data‘tensor\*\*fortheobservedresponse\(e\.g\.

‘y=pm\.Data\(”chose\_left”,…\);pm\.Bernoulli\(”response”,p=p\_left,observed=y\)‘\)\.

Do\*\*not\*\*wrap,copy,orderiveanewvariablefromtheresponsecontainer

beforepassingitto‘observed=‘–thepipelineintrospectsthegraphto

identifytheresponsecontainer,anditmustbethesamenode\.

Allowedtop\-levelimports:‘numpyasnp‘,‘pymcaspm‘,‘pytensor\.tensoraspt‘\.

Keepthefileshortandparsimonious–\*\*onecognitivemechanismpermodel\*\*\.

Thenumberoffreeparametersandfeaturecolumnsamodelreadsshouldmatchthe

singlehypothesis;amodelthatneedsmanyweightedcuestofitisablend,not

ahypothesis\.

\#\#\#Extendingthefeaturespace\(optional\)

Theprecomputedfeaturecolumnsareorder\-destroyingaggregates:theycannotsee

whereinasequencesomethinghappens,thespecificsub\-sequencesitcontains,or

recency\.Ifyourhypothesisdependsonsuchanaspectoftherawsequence,do

\*\*not\*\*trytoforceitfromtheexistingcolumns–derivetheexactstatistic

yourhypothesisneedsbyaddingamodule\-levelfeaturizerto‘candidate\.py‘:

‘‘‘python

defcompute\_features\(sequence\_a:str,sequence\_b:str\)\-\>dict:

”””Returnnewnumericfeaturecolumnsforonestimuluspair\.”””

…

‘‘‘

Thepipelinecallsitontheraw‘sequence\_a‘/‘sequence\_b‘stringsforevery

trialandexposeseachreturnedkeyasanewcolumnyoureadwithamatching

‘pm\.Data‘\.Usethistoexpressahypothesistheprecomputedfeaturescannot–for

example,whetherthe\*last\*tossofeachsequenceisheads\(arecencycue\):

‘‘‘python

defcompute\_features\(sequence\_a,sequence\_b\):

defends\_heads\(seq\):

return1\.0ifseq\.strip\(\)\.upper\(\)\.endswith\(”H”\)else0\.0

return\{”ends\_heads\_a”:ends\_heads\(sequence\_a\),”ends\_heads\_b”:ends\_heads\(sequence\_b\)\}

\#…theninsidethemodel:

\#ends\_heads\_a=pm\.Data\(”ends\_heads\_a”,np\.zeros\(1,dtype=”float64”\)\)

‘‘‘

Rulesfor‘compute\_features‘:itmustreturnadictof\*\*finitenumbers\*\*with

the\*\*samekeysforeverysequencepair\*\*,andthosekeysmustbe\*\*newnames\*\*

\(notcollisionswithexistingcolumns\)\.Itisstill\*\*onehypothesis\*\*–addonly

thefeature\(s\)thesinglemechanismneeds,notagrab\-bagofcuestofitbetter\.

\#\#\#Numericalsafety\(required\)

Yourmodelmustevaluatetoa\*\*finite\*\*log\-probability–amodelwhose‘p\_left‘

orlikelihoodisNaNor‘\-inf‘isrejected\(itwouldcrashMCMCatits

start\-valuecheck\)\.So:

\-Keep‘p\_left‘strictlyinside‘\(0,1\)‘\.A‘sigmoid‘/‘softmax‘alreadydoes

this;ifyoubuildaprobabilityanotherway,clampitwith

‘pt\.clip\(p,1e\-6,1\-1e\-6\)‘\.

\-Use‘pt\.abs\(x\)‘forabsolutevalue–\*\*not\*\*‘pt\.sqrt\(x\*\*2\)‘,whichreturns

NaNinPyTensorforsomeinputs\.

\-Avoid‘log\(0\)‘,divisionbyzero,andunboundedexponentialsoflargescores\.

\#\#Exampleskeleton

‘‘‘python

importnumpyasnp

importpymcaspm

importpytensor\.tensoraspt

withpm\.Model\(\)asmodel:

\#Stimulusinputs–namesmatchresponsesCSVcolumns\.

n\_a=pm\.Data\(”n\_a”,np\.zeros\(1,dtype=”int64”\)\)

h\_a=pm\.Data\(”h\_a”,np\.zeros\(1,dtype=”int64”\)\)

n\_b=pm\.Data\(”n\_b”,np\.zeros\(1,dtype=”int64”\)\)

h\_b=pm\.Data\(”h\_b”,np\.zeros\(1,dtype=”int64”\)\)

\#Freecognitiveparameterwithaprior\(inferencefitsit\)\.

tau=pm\.HalfNormal\(”tau”,sigma=1\.0\)

log\_p\_a=h\_a\*pt\.log\(0\.5\)\+\(n\_a\-h\_a\)\*pt\.log\(0\.5\)

log\_p\_b=h\_b\*pt\.log\(0\.5\)\+\(n\_b\-h\_b\)\*pt\.log\(0\.5\)

p\_left=pm\.Deterministic\(”p\_left”,pm\.math\.sigmoid\(tau\*\(log\_p\_a\-log\_p\_b\)\)\)

\#Observedresponse:thepm\.Datatensorispasseddirectlytoobserved=\.

chose\_left=pm\.Data\(”chose\_left”,np\.zeros\(1,dtype=”int64”\)\)

pm\.Bernoulli\(”response”,p=p\_left,observed=chose\_left\)

‘‘‘

\#\#Self\-check

Beforestopping,confirmbothfilesexist–‘hypothesis\.md‘\(non\-empty\)anda

‘candidate\.py‘thatimportsandexposesa‘pm\.Model‘\.Acandidatewithno

‘hypothesis\.md‘isrejected\.

‘‘‘bash

test\-shypothesis\.md&&echo”hypothesis\.mdOK”

python3\-c”

frompathlibimportPath

fromsrc\.models\.pymc\_inferenceimportload\_pymc\_model,observed\_response\_data

m=load\_pymc\_model\(’candidate’,Path\(’\.’\)\)

print\(’observed:’,observed\_response\_data\(m\)\)

”

\\promptfont