@danielwurgaft: New preprint! AI agents have shown impressive scientific automation capabilities. Can we apply them to psychology resea…

New preprint! AI agents have shown impressive scientific automation capabilities. Can we apply them to psychology research, including human data collection?

Introducing auto-psych, a framework that proposes cognitive models and uses them to design and run human experiments. 1/ https://t.co/dfWrfZrnhO

Incredibly excited about this direction: automating scientific discovery in cognitive science, with agents designing targeted experiments, collecting human data, and using the results to refine their theories. We show that this loop can lead to models that better predict human behavior.

Glad to have been a part of it!

Prior work on automated scientific discovery has made a lot of progress on critiquing and improving models, but these systems are bottlenecked on collecting data.

In psychology, though, lots of data collection happens through online experiments. 2/

We ran auto-psych on a classic cognitive psychology task: subjective randomness.

In the task, participants saw pairs of sequences of coin flips and judged which one felt more random. 3/

A key part of our framework is two nested loops:

• an outer loop for designing and running experiments.
• an inner loop for critiquing and improving cognitive models.

Once an experiment has been run, the system iterates on the data to improve its models. 4/

We first tested whether auto-psych could recover true cognitive models by observing data generated by these models.

Our model recovery simulations recovered both “seed” models inspired by prior work or “alien” models designed to behave in non-human ways. 5/

Next, we let the system gather real human data by running experiments on Prolific! We ran three independent replicates of auto-psych, each of which posted three experiments sequentially.

By the end, each found a model that beat the seed models, with zero human intervention! 6/

We believe that auto-psych, and systems like it, unlock the ability to explore large spaces of possible cognitive models. But we’re not suggesting that humans be taken out of the scientific loop! 7/

Rather than replacing psychologists, systems like auto-psych can let us work at a higher level of abstraction, rapidly considering and evaluating computational theories. 8/

This work was led by Ben Prystawski (who is too wise to be on this platform, but find him on ), and awesome collaborators: @kushin_m, @LinasNasvytis, @michaelyli_ , @noahdgoodman, @mcxfrank

You can read the preprint here: https://arxiv.org/abs/2606.26460 9/