AI can’t simulate human preferences - new study tests LLMs against thousands of real users

Reddit r/ArtificialInteligence 07/07/26, 07:20 PM Papers

llm-simulation human-preferences synthetic-users research choice-tasks evaluation ai-limitations

Summary

A new study tests LLMs across 28 real-world studies and finds they match human majority only 53% of the time, no better than random, challenging the trend of using LLMs to replace human feedback.

https://arxiv.org/abs/2605.18311 There’s a massive trend right now where companies are trying to replace real human feedback with LLM-driven "synthetic users." The idea sounds great on paper - why would you spend money and time recruiting real people to test products, pick design choices, or evaluate options when you can just prompt? They tested LLMs across 28 real-world studies spanning 78 choice tasks to see if their selections matched thousands of actual human participants. The result? The LLMs matched the human majority only 53% of the time. Since most tasks were a choice between two options, that's pretty much same as flipping a coin. Even worse for the "simulation" argument: adding detailed personas and chain-of-thought reasoning yielded practically no improvement. It actually made the semantic similarity to real human justifications worse because the model's "reasoning" just homogenized the outputs and failed to capture actual lived experiences. It looks like LLMs are just trained to replicate what we like about their outputs rather than making them capable of predicting human preferences. Is it time to admit that LLM simulation has hit a hard wall when it comes to replicating human choice?

Original Article

Similar Articles

Evaluating LLMs as Human Surrogates in Controlled Experiments

arXiv cs.CL

This paper evaluates whether off-the-shelf LLMs can reliably simulate human responses in controlled behavioral experiments by comparing LLM-generated data with human survey responses on accuracy perception. The findings show that while LLMs capture directional effects and aggregate belief-updating patterns, they do not consistently match human-scale effect magnitudes, clarifying when synthetic LLM data can serve as behavioral proxies.

Stanford tested 11 LLMs on ~12,000 social situations: they affirm the user 49% more often than humans do

Reddit r/ArtificialInteligence

A Stanford study tested 11 LLMs on about 12,000 social situations and found AI affirms user actions 49% more often than humans, leading to decreased prosocial intentions and increased dependence.

AI is more likely than humans to form biases when hiring

MIT Technology Review

New research shows that LLMs can develop their own biases from experience and stereotype job applicants more than humans, raising concerns about AI in hiring.

@yoheinakajima: there's an increasing amount of research replacing LLMs as human subjects, so I was curious how well this actually work…

X AI KOLs Following

Yohei Nakajima discusses a new paper testing whether LLMs can replace human subjects in behavioral experiments, finding that a GPT-4.1 persona panel passed coarse marginal checks but failed to provide precise treatment-response estimates, so human substitutability is not established.

Using LLMs

Reddit r/singularity

The article reflects on the limited use and understanding of LLMs such as ChatGPT among personal acquaintances, raising questions about widespread AI tool adoption.