When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

# A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses This manuscript is currently under peer review.
Source: [https://arxiv.org/html/2607.26348](https://arxiv.org/html/2607.26348)
Zihan Chen Stevens Institute of Technology Hoboken, NJ, USA zchen61@stevens\.edu &Di Zhu Stevens Institute of Technology Hoboken, NJ, USA dzhu1@stevens\.edu &Lei Nico Zheng University of Massachusetts Boston Boston, MA, USA lei\.zheng@umb\.edu

###### Abstract

Large language models \(LLMs\) are increasingly used as synthetic users, stand\-ins for human respondents whose simulated answers feed product, policy, and market decisions\. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic\-user systems\. A single protocol, run across four models spanning two families and an 8B\-to\-frontier capability range, is applied to two independent domains of real human\-response data: U\.S\. general social attitudes \(General Social Survey\) and cross\-cultural values \(World Values Survey\)\. Every model is benchmarked against a suite of non\-LLM baselines fit on held\-out human data\. Under demographic prompting and the survey\-simulation protocols we test, two failures replicate across both domains, all four models, and both families\. First, at the individual level no LLM beats even the strongest baseline; on cross\-cultural values every model falls well below it, and the gap survives distance\-aware and proper scoring\. Second, models systematically*over\-determine*demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question–group combination and robust to a coding\-invariant measure\. Neither failure is remedied by a larger, more capable model\. A decision\-impact analysis shows why this matters in practice: on a segment\-targeting task the models inflate between\-segment gaps two to fourfold, would direct a team to the wrong segment in half of U\.S\. and most cross\-cultural cases, and manufacture segment splits that do not exist in real people\. We make the cross\-domain benchmark and a validation framework available on request, so that teams relying on synthetic\-user evidence can identify, in advance, the regimes in which it is and is not safe for decision support\.

*K*eywordslarge language models⋅\\cdotsynthetic users⋅\\cdotsurvey simulation⋅\\cdotsocial simulation⋅\\cdotevaluation benchmark⋅\\cdotstereotyping⋅\\cdotresponsible AI

## 1Introduction

Running a human study is slow and expensive, so a fast\-growing body of work proposes replacing human participants with large language models \(LLMs\) prompted to behave as*synthetic users*, using them to pilot surveys, pre\-test questionnaire items, estimate opinion distributions, and stand in for study participants\. The appeal is obvious: a survey that takes months and thousands of dollars to field can, in principle, be approximated in minutes at the cost of a few API calls\.

These simulated respondents are increasingly wired into*decision\-support*pipelines\. Product teams query synthetic personas to prioritize features, marketers use them to pre\-test messaging and estimate segment\-level demand through “silicon samples”\[[34](https://arxiv.org/html/2607.26348#bib.bib12),[21](https://arxiv.org/html/2607.26348#bib.bib13)\], and policy and organizational analysts use them to gauge how a population might react before committing resources\. In each case an intelligent system is asked to manufacture evidence that a human sample would normally provide, and a downstream decision is made on that evidence\. Viewed this way, a synthetic user is an evidence\-producing component of an intelligent decision\-support system, and, like any such component, it should be verified and validated against its intended use before it is trusted\[[37](https://arxiv.org/html/2607.26348#bib.bib37),[27](https://arxiv.org/html/2607.26348#bib.bib40)\]\. This is precisely the setting where an unnoticed failure is costly: if the synthetic users are wrong in a structured way, the resulting decisions inherit that error at scale111“Fail” is scoped throughout to*demographic prompting under survey\-simulation protocols*: the specific elicitation setting we test\. We make no claim about synthetic users produced by other means \(e\.g\. fine\-tuning or richer persona construction\), which the protocol is designed to evaluate in turn\.\. A practical evaluation framework, one that tells a team*in advance*whether synthetic\-user evidence is trustworthy for the decision at hand, is therefore as important as the simulation method itself\. Building that framework is the goal of this paper\.

The central unresolved question is not whether LLMs can produce survey\-*shaped*output \(they trivially can\) but*when the substitution is valid and when it fails*\. The evidence so far is conflicting: some studies report encouraging aggregate alignment\[[2](https://arxiv.org/html/2607.26348#bib.bib2)\], while others warn that substitution is unsafe\[[5](https://arxiv.org/html/2607.26348#bib.bib3),[14](https://arxiv.org/html/2607.26348#bib.bib15)\]\. Prior work has also largely answered the question one domain at a time \(political surveys, or cultural values, or a single national survey\), often reporting a single aggregate similarity number that can look reassuring even when the underlying simulation is poor\. Two questions are therefore under\-answered\. \(i\) Is a synthetic user actually more informative than the trivial thing it is supposed to replace, a lookup of “what do people with these demographics usually answer”? \(ii\) Do conclusions about validity*transfer across domains*, or is each positive result an artifact of one survey, one model, or one prompt?

We address both with a deliberately unified design\. We fix one protocol \(the same models, prompts, decoding, sampling, and metrics\) and apply it to two independent domains of*real*human\-response ground truth:

- •U\.S\. general social attitudes\.The General Social Survey \(GSS\), 2016–2024 waves, 10 attitude questions spanning wellbeing, social trust, moral/policy positions, gender\-role attitudes, and institutional confidence\.
- •Cross\-cultural values\.The World Values Survey \(WVS\) Wave 7, 63 countries, 16 ordinal value questions spanning economic values, corruption perceptions, immigration attitudes, views of science, religiosity, and moral justifiability\.

Across these domains we evaluate four models from two families across an 8B\-to\-frontier capability range, under two prompt formats \(a single\-answer prompt and a probability\-distribution prompt\)\. The design choice that makes our individual\-level results interpretable, and which much prior work omits, is an explicitnaive demographic baseline: for each question we fit the conditional distribution of human answers given demographics on a held\-out portion of the real data, and score every LLM against it on the same respondents\. Without this yardstick, “the model predicts individuals withX%X\\%accuracy” is uninterpretable, because individual survey answers are not a deterministic function of demographics; there is an irreducible ceiling that a trivial predictor already captures\.

#### Research questions

We formalize the study as five research questions\. The first three ascend the levels of analysis at which synthetic users are actually deployed, from the individual respondent \(RQ1\), through the population aggregate \(RQ2\), to the demographic structure of opinion \(RQ3\); the final two then ask whether those answers are stable to modelling choices \(RQ4\) and whether they transfer across domains \(RQ5\)\.

1. RQ1*Individual fidelity vs\. a trivial baseline\.*Do LLM synthetic users predict individual human answers more accurately than a naive demographic\-conditional predictor?
2. RQ2*Aggregate fidelity\.*Do LLM synthetic users reproduce the population\-level distribution of human answers?
3. RQ3*Subgroup structure\.*Do LLMs represent the demographic structure of attitudes faithfully, or do they distort how predictive demographics are of answers?
4. RQ4*Stability and capability\.*Are these behaviors stable across model family, model capability, and output format?
5. RQ5*Cross\-domain transfer\.*Do the answers to RQ1–RQ4 hold in both domains, or are they domain\-specific?

#### Headline findings

Our central result is that, under demographic prompting and the survey\-simulation protocols we test, LLM synthetic users exhibit two distinct failures that replicate across*both*domains,*all four*models, and*both*families\. The first is alack of individual\-level advantage\.As yardsticks we build a suite of non\-LLM predictors, from a “demographic lookup table” \(for any set of demographics, it returns the answer most common among real people with those demographics\) up to learned demographic models \(logistic regression, random forest\)\. On GSS \(U\.S\. attitudes\) every LLM, at best, only ties the lookup table and trails the learned baseline; on WVS \(cross\-cultural values\) every model is*11 to 22 percentage points less accurate*than the baseline\. The gap is not an artifact of exact\-match scoring: it persists under distance\-aware metrics that give partial credit for near\-misses on ordinal scales, and under proper scoring of the distribution outputs, where the model assigns*lower*probability to the true human answer than the baseline\. In short, an LLM asked to role\-play an individual adds no information beyond what the demographics alone already imply, and for value questions it is substantially worse\. The second failure isdemographic over\-determination, or stereotyping\. We measure how strongly a demographic attribute predicts a person’s answer \(the share of the variation in answers that is explained by which group the person is in\) and compare that share for real humans versus for the model\. Models consistently exaggerate it\. For U\.S\. political leaning and confidence in banks, for example, a person’s politics explains only about1\.5%1\.5\\%of the variation in real answers, yet the model behaves as though it explains up to roughly67%67\\%\. This exaggeration is present, and statistically distinguishable from chance, for nearly every question–group combination in both domains, and it replicates under a coding\-invariant association measure that makes no numeric assumption about the answer scale\. Crucially,neither failure is fixed by using a bigger, more capable model: the frontier models stereotype at least as strongly as the small 8B model, and often more\.

#### Contributions

Taken together, our contributions are both methodological and practical\. On the*methodological and empirical*side we make four\. First, we build a compact, reproduciblecross\-domain benchmarkfor LLM synthetic users, spanning U\.S\. social attitudes and 63\-country values under a single protocol and packaged as a reusable evaluation toolkit available on request \(data\-build, inference, and analysis scripts, with all prompts, parser rules, and baselines specified in the appendices\)\. Second, we develop abaseline\-anchored evaluation: we show that the standard individual\-accuracy number is uninterpretable without non\-LLM baselines, and that against a suite of them \(question\-marginal, demographic lookup, and learned demographic models\) LLMs show no individual\-level advantage, a result we confirm with distance\-aware metrics for ordinal scales and proper scoring \(log\-loss, Brier\) for distribution outputs, with paired\-bootstrap confidence intervals throughout\. Third, we introduce astereotyping index, a single, bounded number that compares how predictive a demographic attribute is of the answer in the model versus in real people, reported with confidence intervals and paired with a coding\-invariant companion measure \(Cramér’s V\) that requires no numeric coding of answers; it reframes the common worry that “LLMs flatten minorities” into its measurable opposite for identity\-linked attitudes, namely that LLMs*over\-associate*demographics with answers, treating identity as more decisive than it really is\. Fourth, we provide adecision\-impact analysisthat carries the over\-determination finding through to the decision a synthetic user is meant to support: on the canonical segment\-targeting task, we quantify how far the models inflate between\-segment gaps \(two to fourfold\), how often they would send a team to the wrong segment \(5050–72%72\\%of cases\), and how often they manufacture a segment split that does not exist in real people \(up to41%41\\%of cases on cross\-cultural values\)\.

On the*practical, applied\-intelligent\-systems*side, these pieces combine into our main deliverable: avalidation framework for intelligent synthetic\-user systemsused in decision support\. It is not merely a benchmark of model outputs but a verification\-and\-validation procedure a practitioner runs*before*deployment, one that reports individual and aggregate fidelity separately, always against non\-LLM baselines; reports subgroup determinism and its decision impact; reports invalid\-output rates; and never assumes a larger model is a safer synthetic user\. The framework is the deliverable; the negative findings are the evidence that a framework of this kind is necessary\.

The remainder of the paper reviews related work \(Section[2](https://arxiv.org/html/2607.26348#S2)\), details the data, protocol, and metrics \(Section[3](https://arxiv.org/html/2607.26348#S3)\), presents results by research question \(Section[4](https://arxiv.org/html/2607.26348#S4)\), and discusses implications and limitations \(Sections[5](https://arxiv.org/html/2607.26348#S5)–[6](https://arxiv.org/html/2607.26348#S6)\)\.

## 2Related Work

Our study sits at the intersection of six literatures\. We review each in turn, then state the gap that motivates a baseline\-anchored, cross\-domain validation framework\.

### 2\.1LLMs as simulated survey respondents and participants

A fast\-growing line of work treats LLMs as programmable populations, prompting them with personas or demographics to predict opinions and behaviors\. The optimistic anchor isArgyleet al\.\[[2](https://arxiv.org/html/2607.26348#bib.bib2)\], who show that demographically conditioned prompts can reproduce aggregate patterns in U\.S\. political survey data, launching the “silicon sample” idea; related work extends the substitution to behavioral experiments\[[1](https://arxiv.org/html/2607.26348#bib.bib8)\], economic agents\[[19](https://arxiv.org/html/2607.26348#bib.bib9)\], and the generation of synthetic research data in HCI and product settings\[[17](https://arxiv.org/html/2607.26348#bib.bib11)\]\. A skeptical counter\-current quickly followed\.Bisbeeet al\.\[[5](https://arxiv.org/html/2607.26348#bib.bib3)\]show that LLM\-generated survey responses can misrepresent population and subgroup variation;Santurkaret al\.\[[33](https://arxiv.org/html/2607.26348#bib.bib6)\]find that model “opinions” align more with some demographic groups than others rather than neutrally representing a population; andDominguez\-Olmedoet al\.\[[13](https://arxiv.org/html/2607.26348#bib.bib7)\]show that model survey answers are sensitive to ordering and labeling, and so should not be read as direct measurements of population beliefs\. Position pieces in cognitive science and psychology sharpen the caution: replacing human participants risks confusing linguistic plausibility with validity\[[12](https://arxiv.org/html/2607.26348#bib.bib10),[14](https://arxiv.org/html/2607.26348#bib.bib15),[23](https://arxiv.org/html/2607.26348#bib.bib16)\]\. The practical problem for this paper is precisely this gap between survey\-shaped output and valid evidence\.

### 2\.2Survey\-simulation benchmarks and cross\-cultural value modeling

A parallel effort builds benchmarks that make survey and cross\-cultural value simulation measurable at scale\. WorldValuesBench derives a large demographic\-context\-to\-answer benchmark from the World Values Survey\[[40](https://arxiv.org/html/2607.26348#bib.bib1)\]; SocioBench evaluates LLMs as survey\-response predictors across countries and sociological domains\[[39](https://arxiv.org/html/2607.26348#bib.bib4)\]; and distribution\-specialization work fine\-tunes models to match country\-level response distributions, improving group\-level prediction while still struggling to generalize\[[10](https://arxiv.org/html/2607.26348#bib.bib5)\]\. One of our two domains reuses the WorldValuesBench question set and scales for comparability\. Our work differs from these benchmarks on three axes that matter for a validity claim\. First,*reference point*: WorldValuesBench and SocioBench score model accuracy against the human answer, but not against a non\-LLM predictor of that answer, so a reported accuracy cannot be read as evidence that the simulation adds information; our demographic\-lookup and learned baselines supply exactly that missing reference, and against it the individual\-level advantage disappears\. Second,*level of analysis*: these benchmarks \(and the distribution\-specialization work ofCaoet al\.\[[10](https://arxiv.org/html/2607.26348#bib.bib5)\], which improves group\-level fit through fine\-tuning\) target aggregate or country\-level distributions, which our RQ2 confirms models reproduce reasonably; we show that aggregate success coexists with individual\-level failure and with a subgroup distortion the aggregate metric cannot see\. Third,*cross\-domain transfer*: each benchmark fixes a single survey instrument, so a positive or negative result could be an artifact of that instrument; by holding one protocol fixed across a U\.S\. attitude survey and a 63\-country value survey we can separate properties of LLM simulation from properties of a dataset\. In short, we complement calibrated value and persona benchmarks by asking whether simulated users add individual\-level information beyond a demographic baseline, whether the demographic structure of answers is faithful, and whether the answer transfers across domains\.

### 2\.3Persona prompting, silicon samples, and prompt artifacts

Persona and demographic prompting is the dominant mechanism for constructing synthetic users, and it is used well beyond political surveys: in marketing and consumer research, where “silicon samples” are proposed as low\-cost respondent pools\[[34](https://arxiv.org/html/2607.26348#bib.bib12)\], and in automated perceptual analysis for product decisions\[[21](https://arxiv.org/html/2607.26348#bib.bib13)\]\. Crucially, persona assignment is not a neutral conditioning operation\. It can amplify toxicity and stereotypes\[[11](https://arxiv.org/html/2607.26348#bib.bib21)\], the surface architecture of a prompt can induce methodological artifacts\[[7](https://arxiv.org/html/2607.26348#bib.bib14)\], sociodemographic persona formulations materially change model behavior\[[24](https://arxiv.org/html/2607.26348#bib.bib23)\], and persona generation itself carries quality and validity pitfalls\[[20](https://arxiv.org/html/2607.26348#bib.bib22)\]\. This literature motivates treating prompt\-surface stability and demographic over\-determination \(our RQ3–RQ4\) as first\-order validity conditions rather than minor robustness checks\.

### 2\.4Social simulation and LLM agents

Synthetic survey respondents are one instance of a broader movement to simulate humans with LLMs\. Generative agents endow LLMs with memory, reflection, and planning to produce believable individual and social behavior\[[28](https://arxiv.org/html/2607.26348#bib.bib18)\], building on earlier populated prototypes for social computing systems\[[29](https://arxiv.org/html/2607.26348#bib.bib17)\]; recent surveys map the field from individual agents to whole\-society simulation\[[26](https://arxiv.org/html/2607.26348#bib.bib20)\]and assess the promise and epistemic risks of LLMs for computational social science\[[41](https://arxiv.org/html/2607.26348#bib.bib19)\]\. This work mostly evaluates believability or downstream system behavior; we contribute a quantitative, human\-anchored validity benchmark for one high\-stakes class of these simulations\.

### 2\.5Algorithmic stereotyping, fairness, and subgroup validity

Our over\-determination finding connects to a long line of work on social bias in learned representations: human\-like biases in embeddings\[[9](https://arxiv.org/html/2607.26348#bib.bib29),[6](https://arxiv.org/html/2607.26348#bib.bib30)\], stereotyped associations in generation\[[36](https://arxiv.org/html/2607.26348#bib.bib32)\], and representational harms of large models\[[4](https://arxiv.org/html/2607.26348#bib.bib33)\]\. The fairness literature further establishes that aggregate performance can mask subgroup harm: intersectional accuracy disparities\[[8](https://arxiv.org/html/2607.26348#bib.bib31)\], group\-conditioned error criteria\[[18](https://arxiv.org/html/2607.26348#bib.bib35)\], hand\-built bias benchmarks for constrained\-choice QA\[[30](https://arxiv.org/html/2607.26348#bib.bib34)\], and the reminder that fairness must be assessed within the sociotechnical context of use\[[35](https://arxiv.org/html/2607.26348#bib.bib36)\]\. Our stereotyping index adapts this subgroup lens but differs in target: rather than asking whether a model attaches a negative label to a group, we compare*how predictive*demographics are of an answer in the model versus in real humans, turning stereotyping into a human\-comparable validity metric\.

### 2\.6Decision\-support validity and expert\-system validation

For an applied\-AI venue, the decisive framing is that synthetic users are evidence\-producing components inside intelligent decision\-support systems, and such components require verification and validation against intended use\. Information\-systems research treats predictive validity \(out\-of\-sample performance against baselines\) as a first\-class evaluation goal distinct from explanatory fit\[[37](https://arxiv.org/html/2607.26348#bib.bib37)\]; the decision\-support and expert\-systems tradition has long emphasized verification, validation, and evaluation of intelligent artifacts\[[3](https://arxiv.org/html/2607.26348#bib.bib38),[27](https://arxiv.org/html/2607.26348#bib.bib40),[32](https://arxiv.org/html/2607.26348#bib.bib39)\]\. Multi\-metric reporting norms from ML evaluation reinforce the same discipline: holistic evaluation across accuracy, calibration, robustness, and fairness\[[22](https://arxiv.org/html/2607.26348#bib.bib24)\], behavioral testing beyond aggregate accuracy\[[31](https://arxiv.org/html/2607.26348#bib.bib25)\], broad capability benchmarks\[[38](https://arxiv.org/html/2607.26348#bib.bib26)\], and standardized documentation of models and datasets\[[25](https://arxiv.org/html/2607.26348#bib.bib27),[15](https://arxiv.org/html/2607.26348#bib.bib28)\]\. Finally, survey methodology’s total\-survey\-error framework distinguishes representation from measurement error\[[16](https://arxiv.org/html/2607.26348#bib.bib41)\], which is exactly why individual, aggregate, and subgroup validity are separate questions: an LLM synthetic user introduces a new error source that can look benign in aggregate while failing at the individual or subgroup level\.

### 2\.7What is missing

This prior work is strong but fragmented: each study typically fixes one domain, one or two models, and one evaluation lens, which makes it hard to tell whether a reported failure \(or success\) is a property of LLM simulation or of the specific setup\. Two gaps matter for practice\. First, individual\-level results are usually reported without*non\-LLM baselines*, so it is unclear whether the model beats a demographic lookup, let alone a learned demographic model\[[37](https://arxiv.org/html/2607.26348#bib.bib37)\], and whether any apparent gap is merely an artifact of harsh exact\-match scoring on ordinal scales\. Second, subgroup analyses typically ask whether groups are “flattened,” but do not quantify, on a human\-comparable scale, how much demographics*should*explain versus how much the model makes them explain\. We close both gaps by anchoring individual fidelity to a baseline suite scored with distance\-aware and proper\-scoring metrics, and by quantifying subgroup determinism with both a variance\-based and a coding\-invariant association measure\. Holding one protocol fixed across two domains and four models, we then test whether the conclusions transfer\.

## 3Data, Protocol, and Metrics

### 3\.1Domains and scope

We use two public human\-response datasets and state the scope we actually use; all claims are restricted to that scope\.

GSS \(U\.S\. social attitudes\)\.We use the GSS 1972–2024 cumulative file restricted to the2016–2024waves\. Recent waves give large per\-subgroup cells and partially mitigate pretraining contamination relative to decades\-old waves\. After removing respondents with missing values on any prompt demographic, the in\-scope pool is 14,704 respondents and 85,898 \(respondent, question\) pairs over 10 questions\. Demographics used in the prompt: age, sex, race, highest degree, region, political views, party identification\.

WVS \(cross\-cultural values\)\.We use WVS Wave 7 \(inverted CSV, v6\.0\)\. We retain the 16 value questions from the WorldValuesBench probe set\[[40](https://arxiv.org/html/2607.26348#bib.bib1)\]that are present in this release; question wordings and ordinal scales follow WorldValuesBench for comparability\. After demographic filtering the in\-scope pool is 91,774 respondents across 63 countries and 1,426,473 \(respondent, question\) pairs\. Demographics: age group, sex, education level, settlement type \(urban/rural\), and country\.

### 3\.2Sampling for measurable distributions

Our headline analyses need a reliable estimate of the human answer*distribution*within each \(question, group\) cell, which requires enough people in each cell\. We therefore sample by cell rather than drawing respondents at random\. For each domain we draw about100100respondents per question \(GSS: 993 rows; WVS: 1,458 rows\), allocated so that the main grouping is well covered \(GSS: degree×\\timesrace; WVS: country\)\. Importantly, the human answer distributions and the demographic baseline are always estimated from the full dataset, never from this small sample, so the human side of every comparison rests on large, stable numbers\.

### 3\.3Models and prompting

We evaluate four chat\-style models through a single, common interface: Claude Haiku 4\.5 and Claude Sonnet 4\.6 \(the*closed*family: proprietary, accessed via an API\) and Llama\-3\.1\-8B and Llama\-3\.3\-70B \(the*open*family: open\-weight and publicly downloadable\)\. Together these span two independent model families and a wide capability range, from a small 8B model to frontier\-scale systems\. Each sampled respondent is run under two prompt formats and two independent generation runs \(“seeds,” which differ only in the model’s internal randomness, letting us gauge run\-to\-run noise\):

- •Style A, the single\-answer prompt\.The model is shown the demographic profile and question and must return one answer \(an option letter for short\-labelled questions, or the scale number for numeric scales\)\.
- •Style C, the distribution prompt\.The model must return a probability for each answer option, expressed as JSON\.

We use these two names \(Style A / single\-answer prompt, Style C / distribution prompt\) interchangeably throughout\. The model’s internal randomness is held at a fixed setting across models; the two seeds probe run\-to\-run stability\. All calls use the same output parser, which flags any response it cannot read as a valid answer as*invalid/refused*\.

### 3\.4Non\-LLM baselines

The individual\-accuracy number is only interpretable against what a non\-LLM predictor achieves on the same people, so we build a suite of four non\-LLM baselines\. All are fit on a held\-out 50% split of the in\-scope pool \(assigned by a hash of the respondent id\) and evaluated on the evaluation\-sample rows, which come from the other split, so no baseline is ever fit and scored on the same respondents\. The four baselines, from simplest to most sophisticated, are as follows \(full formulae and hyperparameters are in Appendix[E](https://arxiv.org/html/2607.26348#A5)\):

- •Question marginal\.The single most common answer to the question among the fit humans \(ignores demographics entirely\)\.
- •Demographic lookup\(our primary yardstick\)\. A predictor of the conditional answer distribution given demographics, which backs off from a fine demographic cell to a coarse cell to the question marginal, so it always yields a prediction\. For WVS the finest cell begins with country; the coarse cell is country alone, which we also report separately as a*country\-only*baseline\.
- •Multinomial logistic regressionon the one\-hot\-encoded prompt demographics, fit per question\.
- •Random foreston the same features \(300 trees\), a nonlinear learned baseline\.

The demographic lookup and logistic model also emit a full predicted*distribution*over answers, which we use as the reference for the proper\-scoring comparison against the model’s distribution outputs \(Section[3\.5](https://arxiv.org/html/2607.26348#S3.SS5)\)\. These baselines are the yardsticks for RQ1: following the predictive\-validity tradition in information systems\[[37](https://arxiv.org/html/2607.26348#bib.bib37)\], a synthetic user earns its place only by beating a simpler predictor on held\-out human data\.

### 3\.5Metrics

We use four families of metric, one per research question\. Reporting individual, aggregate, and subgroup fidelity separately mirrors the total\-survey\-error distinction between measurement and representation error\[[16](https://arxiv.org/html/2607.26348#bib.bib41)\]and the multi\-metric evaluation norms now standard for language models\[[22](https://arxiv.org/html/2607.26348#bib.bib24),[31](https://arxiv.org/html/2607.26348#bib.bib25)\]: a single aggregate number hides exactly the failures we care about\. Each metric is stated in plain terms first, then defined precisely\. Every headline comparison against a baseline carries apaired\-bootstrap 95% confidence interval: we resample the evaluation\-sample rows500500times, scoring model and baseline on the*same*rows in each resample, and report the interval of the difference, so the reader can tell a real margin from sampling noise\.

Individual fidelity \(RQ1\): can the model guess one person’s answer?We report*accuracy*\(the fraction of individuals whose exact answer the model predicts correctly\) next to each baseline’s accuracy on the same people\. For the distribution prompt \(Style C\), the model’s single “guess” is the option it assigned the highest probability \(the arg\-max of its distribution\)\. Exact\-match accuracy is deliberately harsh on long ordinal scales, so for the ordinal questions we also report two*distance\-aware*metrics that give partial credit for being close\. The*mean absolute error*\(MAE\) is the average gap, in scale points, between the single predicted answer and the true one\. The*earth\-mover’s distance*\(EMD\) uses the whole predicted distribution from the distribution prompt: it is the expected number of scale steps needed to move the predicted probability mass onto the true answer, and so rewards a distribution that concentrates near the truth even when its mode is wrong\. Both are reported for the model and for the baseline on the same rows\.

Aggregate fidelity \(RQ2\): does the model reproduce the group’s answer*spread*?Rather than any one person, we compare the overall*share*of people choosing each option \(the answer distribution\) between the model and real humans\. We summarize the gap between two distributions with the*Jensen–Shannon \(JS\) divergence*, a standard0\-to\-11measure of how different two probability distributions are;0means identical, and larger means further apart, so lower is better\. Because we sample by cell rather than at random \(Section[3\.2](https://arxiv.org/html/2607.26348#S3.SS2)\), the raw pooled model distribution reflects our sampling design, not the human population\. We therefore also report a*population\-reweighted*JS, in which each cell’s model distribution is weighted back to that cell’s share of the real population before comparing to the true \(full\-table\) human distribution\. If the two versions agree, the aggregate\-fidelity result is not an artifact of the sampling design\.

For the distribution prompt \(Style C\), arg\-max accuracy discards most of the predicted distribution, so we additionally score the*whole*distribution with two proper scoring rules evaluated at the true human answer: the*log\-loss*\(negative log\-likelihood the model places on the true answer\) and the*Brier score*\(squared error of the predicted distribution against the one\-hot true answer\)\. Lower is better, and both are reported against the baseline’s conditional distribution on the same rows; a proper score is the right way to ask whether the distribution puts mass where the human actually answered\.

Subgroup determinism \(RQ3\): does the model treat identity as destiny?Fairness research has long argued that aggregate performance can hide subgroup harm and must be assessed group by group\[[8](https://arxiv.org/html/2607.26348#bib.bib31),[18](https://arxiv.org/html/2607.26348#bib.bib35)\]\. We adopt that lens but ask a sharper, human\-anchored question: how strongly a demographic attribute \(say, political leaning\)*predicts*the answer, for real people and for the model\. The natural measure is the share of the total variation in answers that is explained by which group a person belongs to\. Statisticians call thisη2\\eta^\{2\}\(“eta\-squared”\), the ratio of the variation*between*groups to the total variation \(between plus within groups\); it runs from0\(the attribute tells you nothing about the answer\) to11\(the attribute determines the answer completely\)\. We computeη2\\eta^\{2\}the same way for humans and for the model\. Our headline measure, thestereotyping index, is simply the difference,

Δ​η2=ηmodel2−ηhuman2,\\Delta\\eta^\{2\}\\;=\\;\\eta^\{2\}\_\{\\text\{model\}\}\-\\eta^\{2\}\_\{\\text\{human\}\},which is positive when the model treats a demographic attribute as*more*predictive of the answer than it actually is among real people, that is, when the model stereotypes\. Because it is a difference of two bounded quantities computed identically, it is not distorted by which prompt style produced the model’s answers, and it captures the full spread within each group rather than only the group averages\. We attach a95%95\\%confidence interval to each value by resampling respondents500500times \(a bootstrap\), so we can tell a real effect from sampling noise\.

η2\\eta^\{2\}treats the answer codes as numbers, which is natural for the ordinal questions but would be an arbitrary choice for nominal ones\. To guard against the objection that the finding depends on how answers are coded, we add a*coding\-invariant*companion measure that uses no numeric codes at all: Cramér’s V, the standard association strength between two categorical variables \(here, group membership and the chosen answer\), computed identically for humans and for the model from the group×\\timesanswer contingency table\. Its stereotyping analogue isΔ​V=Vmodel−Vhuman\\Delta V=V\_\{\\text\{model\}\}\-V\_\{\\text\{human\}\}, again with a bootstrap95%95\\%CI\. Because it is purely nominal, we can report it for*all*questions, including the nominal GSS items that theη2\\eta^\{2\}index omits\. If both the numericΔ​η2\\Delta\\eta^\{2\}and the coding\-freeΔ​V\\Delta Vare positive, over\-determination is not an artifact of the answer coding\.

Decision impact: what does over\-determination cost a decision that acts on it?The stereotyping index measures the distortion in the model’s world; a decision\-support venue needs to know what that distortion does to a decision\. We therefore translate it into the terms of the single most common use of a synthetic user in practice:*segment targeting*\. A product, marketing, or policy team asks which demographic segment is most extreme on an attitude, and how large the between\-segment gap is, in order to target the top segment, size a niche, or claim “this attitude splits along <segment\>\.” For each \(question, demographic axis\) we compute, identically for the model and for real humans, the*segment share*pgp\_\{g\}\(the fraction of segmentggon the high end of the answer scale\), and read three quantities a targeting decision keys on: the*between\-segment gap*maxg⁡pg−ming⁡pg\\max\_\{g\}p\_\{g\}\-\\min\_\{g\}p\_\{g\}; the*target segment*arg⁡maxg⁡pg\\arg\\max\_\{g\}p\_\{g\}; and, when the model’s target differs from the human target, the*targeting cost*\(the attitude a team forgoes, in real human share, by acting on the model’s segment instead of the true one\)\. From these we report \(i\) the*gap\-inflation factor*gapmodel/gaphuman\\text\{gap\}\_\{\\text\{model\}\}/\\text\{gap\}\_\{\\text\{human\}\}, how much the model exaggerates the between\-segment difference; \(ii\) the*wrong\-target rate*, the fraction of \(question, axis\) pairs where the model would send a team to the wrong segment; and \(iii\) the*spurious\-split rate*, the fraction where humans show a negligible gap \(≤0\.10\\leq\\\!0\.10\) but the model shows a large one \(≥0\.25\\geq\\\!0\.25\), a segment split a team would “discover” and act on that does not exist in real people\. Human shares use the full in\-scope table; the model gap carries a95%95\\%bootstrap CI over respondents\.

Stability \(RQ4\): do the conclusions survive irrelevant changes?We report the rate of invalid or refused outputs \(answers the parser cannot read as a valid choice\), agreement between two decoding runs, and how results move across model family, model size, and prompt wording\.

### 3\.6Estimation choices and their robustness

Four choices in the estimation deserve explicit justification, since each is a place a reader might worry the headline effects are an artifact of the analysis rather than of the models\. We resolve each on the data itself and report the checks in Section[4\.5](https://arxiv.org/html/2607.26348#S4.SS5)and Appendix[F](https://arxiv.org/html/2607.26348#A6)\.

*Confidence level\.*We use95%95\\%intervals throughout\. The individual\-level deficit \(RQ1\) and the over\-determination effect \(RQ3\) are large relative to their uncertainty, so moving from a90%90\\%to a stricter95%95\\%interval leaves every headline unchanged: on WVS every individual\-accuracy margin remains strictly below zero, on GSS every model remains a statistical tie\-or\-worse against the baseline, and the stereotyping index remains significant for the large majority of question–group pairs \(Table[1](https://arxiv.org/html/2607.26348#S4.T1)\)\.

*Multiple comparisons\.*The stereotyping index is tested over many \(question, axis\) pairs, so a per\-pair interval does not control the false\-discovery rate\. We therefore also apply a Benjamini–Hochberg FDR correction atq=0\.05q=0\.05across all pairs within a model and prompt style\. Because the effects are large, correction barely changes the count of significant pairs \(e\.g\. Sonnet single\-answer:32/32→32/3232/32\\to 32/32on WVS,10/12→10/1210/12\\to 10/12on GSS; Appendix[F](https://arxiv.org/html/2607.26348#A6)\), so “significant for the majority of pairs” is not an artifact of uncorrected multiplicity\.

*Respondent clustering\.*A minority of respondents answer more than one question \(7\.6%7\.6\\%on GSS,1\.8%1\.8\\%on WVS\), so rows are not fully independent\. We recompute the RQ1 margin with a*cluster*bootstrap that resamples respondents rather than rows; the95%95\\%intervals are essentially unchanged and every WVS deficit still excludes zero \(Appendix[F](https://arxiv.org/html/2607.26348#A6)\)\.

*Survey weights\.*GSS and WVS ship official design/population weights \(wtssps,W\_WEIGHT\)\. We estimate human targets unweighted, and justify that choice by recomputing, with and without weights on the full in\-scope table, the two human\-side quantities the claims rest on: the population answer distribution \(the RQ2 target\) and the humanη2\\eta^\{2\}\(the RQ3 baseline that the stereotyping index subtracts\)\. Weighting moves the human distribution by a mean Jensen–Shannon divergence of≤0\.0003\\leq\\\!0\.0003and the humanη2\\eta^\{2\}by at most0\.0070\.007, both one to three orders of magnitude smaller than the effects we report \(JS gaps of0\.040\.04–0\.380\.38; stereotyping indices of\+0\.05\+0\.05to\+0\.7\+0\.7\)\. Weighting therefore cannot account for either failure, so we report the simpler unweighted analysis and document the check in Appendix[F](https://arxiv.org/html/2607.26348#A6)\.

## 4Results

Table[1](https://arxiv.org/html/2607.26348#S4.T1)reports the full cross\-model, cross\-domain results, and Table[2](https://arxiv.org/html/2607.26348#S4.T2)the RQ1 robustness checks \(stronger baselines, distance\-aware and proper scoring\)\. The individual\-fidelity result is shown in Figures[1](https://arxiv.org/html/2607.26348#S4.F1)and[2](https://arxiv.org/html/2607.26348#S4.F2)\(the latter adding paired\-bootstrap confidence intervals on the accuracy margin over the baseline\), and the over\-determination result in Figures[3](https://arxiv.org/html/2607.26348#S4.F3)and[4](https://arxiv.org/html/2607.26348#S4.F4)\(the latter resolving it to the individual\-question level\)\.

Table 1:Cross\-domain, cross\-model results\.Δ\\Deltabase==individual accuracy minus the demographic\-lookup baseline \(negative==worse than the baseline\); paired\-bootstrap95%95\\%CIs for this margin are shown per model in Figure[2](https://arxiv.org/html/2607.26348#S4.F2), and the stronger baselines, distance\-aware, and proper\-scoring comparisons in Table[2](https://arxiv.org/html/2607.26348#S4.T2)\. JS==mean Jensen–Shannon divergence \(lower is better; population\-reweighted values agree, see Section[4\.2](https://arxiv.org/html/2607.26348#S4.SS2)\)\.Δ​η2¯\\overline\{\\Delta\\eta^\{2\}\}==median stereotyping index \(positive==over\-determination\); “sig”==number of question–group pairs whose 95% CI excludes zero, out of the total \(a coding\-invariant Cramér’s V replication is in Section[4\.3](https://arxiv.org/html/2607.26348#S4.SS3)\)\. Style A==single answer, Style C==distribution\.†Style\-C cells with elevated invalid rates \(Section[4\.5](https://arxiv.org/html/2607.26348#S4.SS5)\); interpret with caution\. Llama\-8B Style\-C on WVS is effectively unusable \(85% invalid,n=440n=440\)\.

### 4\.1RQ1: LLMs do not beat, and on values fall well below, any non\-LLM baseline

On GSS, the individual accuracy of every model is at or below the demographic lookup baseline \(0\.5890\.589\)\. The best margin any model achieves is−0\.001\-0\.001\(Sonnet, Style C\), a paired\-bootstrap tie \(CI\[−0\.028,0\.022\]\[\-0\.028,0\.022\]\); Llama\-8B is99points worse \(0\.4960\.496vs\.0\.5890\.589, CI\[−0\.120,−0\.067\]\[\-0\.120,\-0\.067\]\)\. The lookup is not a weak straw man: a learnedlogisticbaseline reaches0\.6220\.622and a random forest0\.5830\.583\(Table[2](https://arxiv.org/html/2607.26348#S4.T2)\), so the best LLM also trails the strongest non\-LLM predictor\. On WVS the gap is far larger and uniformly negative: under the single\-answer prompt every model is13\.9 to 21\.8 accuracy points belowthe demographic baseline \(e\.g\. Haiku Style A:0\.1700\.170vs\.0\.3880\.388\), and every model under either prompt is below the learned baselines as well\. On WVS every one of these margins has a95%95\\%CI that lies strictly below zero; on GSS no margin is positive, with the deficits for Llama\-8B \(both prompts\) and Sonnet’s single\-answer prompt clearing95%95\\%significance and the remaining cells statistical ties at or below the baseline\. So for predicting an individual’s response, and especially a cross\-cultural value response, a simple demographic predictor beats all four LLMs\.

Two robustness checks close the obvious escape routes\. First, the deficit is not an artifact of harsh exact\-match scoring: on the ordinal questions the model’s*distance\-aware*error is also worse than the baseline’s\. On GSS the model’s MAE is0\.560\.56–0\.720\.72versus0\.540\.54for the baseline; on WVS’s long scales it is1\.831\.83–2\.272\.27versus1\.741\.74\(Table[2](https://arxiv.org/html/2607.26348#S4.T2)\)\. The distributional EMD, which credits a distribution for placing mass near the truth even when its mode is wrong, tells the same story \(0\.760\.76–0\.830\.83vs\.0\.730\.73on GSS;1\.941\.94–2\.302\.30vs\.1\.671\.67–2\.022\.02on WVS\)\. Giving partial credit for near\-misses does not rescue the model\. Second, scoring the*whole*Style C distribution with proper scoring rules tells the same story: the model assigns*lower*probability to the true human answer than the baseline does\. On GSS the model’s log\-loss is0\.840\.84–1\.031\.03versus0\.810\.81for the baseline \(Brier0\.520\.52–0\.620\.62vs\.0\.500\.50\); on WVS1\.761\.76–2\.262\.26versus1\.701\.70\(Brier0\.800\.80–0\.890\.89vs\.0\.710\.71\)\. This directly answers RQ1: synthetic users provide*no*individual\-level advantage over a demographic baseline under any of these scorings, and on values they are actively worse\.

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/GSS/fig_individual_vs_baseline.png)

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/WVS/fig_individual_vs_baseline.png)

Figure 1:Individual accuracy vs\. the naive demographic baseline \(dashed line\), GSS \(left\) and WVS \(right\)\. No model exceeds the baseline on GSS; every model falls well below it on WVS\. The individual\-level failure \(RQ1\) holds in both domains and is more severe for cross\-cultural values\.![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/GSS/fig_delta_base_ci.png)

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/WVS/fig_delta_base_ci.png)

Figure 2:Individual\-accuracy margin over the demographic baseline \(Δbase\\Delta\_\{\\text\{base\}\}\) with paired\-bootstrap95%95\\%confidence intervals, GSS \(left\) and WVS \(right\)\. Negative means worse than the baseline\. Every interval lies below zero except the two GSS ties, and the WVS deficits are large and tight\.Table 2:RQ1 robustness: the individual\-level deficit survives stronger baselines, distance\-aware scoring, and proper scoring\.*Top:*pooled individual accuracy of the four non\-LLM baselines \(the best LLM is shown for reference\)\.*Bottom:*per\-model distance\-aware error on ordinal questions and proper scores, each next to the baseline on the same rows \(all lower is better\): MAE==mean absolute error of the single\-answer prediction; EMD==earth\-mover’s distance of the full predicted distribution \(Style C\) from the true answer, in scale steps; log\-loss and Brier are evaluated on the distribution prompt at the true answer\. Each cell shows model / baseline \(m/b\) on the same rows\. In every cell the model is at best tied and usually worse than the baseline\.

### 4\.2RQ2: Aggregate distributions are reproduced reasonably, especially under Style C

Reproducing the group’s overall answer*spread*is much easier for the models than guessing individuals, and asking the model directly for a distribution \(Style C\) does this markedly better than asking for a single answer \(Style A\)\. Recall that the JS divergence runs from0\(distributions identical to humans’\) upward, so smaller is better\. On GSS, Style C reaches0\.0110\.011–0\.0230\.023\(essentially the human distribution\), versus0\.0560\.056–0\.0900\.090for Style A\. WVS shows the same ordering \(Style C≈0\.037\\approx 0\.037–0\.0460\.046for the models that produce valid output, versus0\.230\.23–0\.380\.38for Style A\)\. This confirms, across both domains, the familiar pattern that a model can mimic “the average group” far better than any individual within it\. This aggregate result is not an artifact of the cell\-based sampling: reweighting each cell back to its human population share leaves the JS divergence essentially unchanged \(e\.g\. Haiku GSS Style C0\.0108→0\.01090\.0108\\to 0\.0109; Sonnet WVS Style C0\.0371→0\.03730\.0371\\to 0\.0373\), so the model is being compared against the true population distribution, not against our sampling design\.

### 4\.3RQ3: LLMs over\-determine demographics \(stereotyping\), not flatten them

A common worry is that LLMs*flatten*groups, washing out the differences between them\. For identity\-linked attitudes we find the opposite\. The stereotyping index is*positive*for essentially every model and prompt style in both domains, and its confidence interval excludes zero for the large majority of question–group combinations \(Table[1](https://arxiv.org/html/2607.26348#S4.T1)\): models treat demographics as*more*decisive of the answer than they are among real people\. The effect is dramatic for politically\- and identity\-linked questions\. On GSS, a person’s political views explain about10%10\\%of the real variation in gender\-role attitudes, but in the models they appear to explain6060–69%69\\%; for confidence in banks, political views explain only about1\.5%1\.5\\%of the real variation yet up to about67%67\\%in the model, roughly a forty\-fold exaggeration\. On WVS, taking country as the grouping, nearly all3232question–country combinations are significantly over\-determined for the two Claude models and the larger Llama\. The simulated population is therefore not a blurred copy of the real one; it is a*caricature*, in which who you are dictates what you think far more tightly than it does in reality\.

This does not depend on how we code the answers\. The coding\-invariant measure, Cramér’s V, which uses no numeric codes and treats every answer as an unordered category, tells the same story on*all*questions, including the nominal GSS items theη2\\eta^\{2\}index omits\. The medianΔ​V\\Delta Vis positive for both Claude models in both domains \(GSS Sonnet Style A\+0\.152\+0\.152, Haiku\+0\.108\+0\.108; WVS Sonnet Style A\+0\.160\+0\.160, Llama\-70B\+0\.166\+0\.166\), and its bootstrap CI excludes zero for a majority of question–group pairs \(e\.g\.2222of4040for Sonnet on GSS,2323of3232on WVS\)\. Over\-determination is thus a property of the model’s behavior, not of the numeric coding of the scale\.

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/GSS/fig_stereotyping_index.png)

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/WVS/fig_stereotyping_index.png)

Figure 3:Stereotyping index \(ηmodel2−ηhuman2\\eta^\{2\}\_\{\\text\{model\}\}\-\\eta^\{2\}\_\{\\text\{human\}\}\) by model and prompt style, GSS \(left\) and WVS \(right\); higher means the model treats demographics as more decisive of the answer than they are among real people\. All reliable bars are positive in both domains: every model over\-determines demographics \(RQ3\), and the larger, more capable models do not reduce the effect \(RQ4\)\. Distribution\-prompt bars with high invalid\-output rates are omitted and marked “output unreliable” \(see Section[4\.5](https://arxiv.org/html/2607.26348#S4.SS5)\)\.![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/GSS/fig_question_stereo.png)

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/WVS/fig_question_stereo.png)

Figure 4:Question\-level over\-determination for the frontier model \(Sonnet 4\.6, single\-answer prompt\) with bootstrap95%95\\%CIs, along the primary grouping axis \(GSS: political views; WVS: country\)\. Over\-determination is not a model\-average artifact: it holds question by question, with confidence intervals well above zero for the strongest items\.
### 4\.4Decision impact: over\-determination produces wrong and spurious segment decisions

Over\-determination is not just a statistical curiosity; it changes what a team would decide\. Reading segment structure off the synthetic users rather than off real humans distorts the targeting decision in three concrete ways \(Figure[5](https://arxiv.org/html/2607.26348#S4.F5)\)\. First, the modelinflates the between\-segment gapa decision keys on: for the frontier model \(Sonnet, single\-answer\) the median gap is2\.3×2\.3\\timesthe true human gap on GSS and2\.5×2\.5\\timeson WVS, and every model inflates it \(medians1\.31\.3–4\.1×4\.1\\timeson GSS,2\.02\.0–4\.7×4\.7\\timeson WVS\)\. A team sizing a segment difference off synthetic users over\-estimates it roughly two to fourfold\. Second, the model oftentargets the wrong segment: itsarg⁡max\\arg\\maxsegment differs from the true humanarg⁡max\\arg\\maxfor50%50\\%of GSS \(question, axis\) pairs and72%72\\%of WVS pairs \(Sonnet single\-answer\), so a team choosing which segment to prioritize would, more often than not on values, pick a segment that is not actually the most extreme\. Third, and most damaging, the modelmanufactures segment splits that do not exist: on WVS,28%28\\%of pairs \(Sonnet single\-answer; up to41%41\\%for Haiku and Llama\-70B\) fall in the spurious\-split regime: humans differ across segments by≤10\\leq\\\!10points while the model differs by≥25\\geq\\\!25\. Concrete cases make the failure vivid\. Among U\.S\. respondents, confidence in banks barely varies with political leaning \(a “great deal” of confidence ranges only1313–24%24\\%across the seven political categories\), yet the model treats politics as roughly67%67\\%determinative of the answer; a team reading the simulation would conclude, wrongly, that trust in banks is sharply polarized\. On WVS, attitudes to government responsibility \(Q108\) differ across education levels by under55points among real people but by7474points in the model \(16×16\\times; CI\[0\.60,0\.86\]\[0\.60,0\.86\]\), a segment split with no basis in the human data\. Because these errors are*structured*, always in the direction of exaggerated group difference, they do not average out across a portfolio of decisions; they systematically push a team toward over\-segmenting a population and over\-targeting the segment the model has caricatured\.

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/GSS/fig_decision_gap.png)

![Refer to caption](https://arxiv.org/html/2607.26348v1/figures/WVS/fig_decision_gap.png)

Figure 5:Decision impact of over\-determination for the frontier model \(Sonnet 4\.6, single\-answer prompt\): the between\-segment gap a targeting decision reads off the model \(vertical axis, with95%95\\%bootstrap CI\) against the true human gap \(horizontal axis\), one point per \(question, demographic axis\), GSS \(left\) and WVS \(right\)\. The dashed line is faithful reading \(model==human\); points above it are gaps the model exaggerates\. Shaded: the*spurious\-split*zone \(human gap≤0\.10\\leq\\\!0\.10but model gap≥0\.25\\geq\\\!0\.25\), the segment differences a team would act on that do not exist in real people\. Almost every point lies above the faithful line\.
### 4\.5RQ4: Capability does not fix it; outputs are unstable to format and prompt surface

Three stability results stand out\. First,capability does not remedy either failure\. Within the closed family, the frontier model \(Sonnet 4\.6\) stereotypes*more*than the smaller Haiku 4\.5 \(medianΔ​η2\\Delta\\eta^\{2\}\+0\.104\+0\.104vs\.\+0\.059\+0\.059on GSS Style C;\+0\.103\+0\.103vs\.\+0\.056\+0\.056on WVS Style C\), and does not beat the baseline on individuals\. Within the open family, Llama\-70B has the highest stereotyping index on WVS \(\+0\.136\+0\.136\)\. Larger, more capable models are therefore not safer synthetic users on these axes\. Second,format stability is a real failure mode for smaller models\. The distribution prompt \(Style C\) over long numeric scales produced high invalid\-output rates: Llama\-8B reached 85% invalid on WVS \(rendering its Style\-C numbers unusable,n=440n=440\) and 57% on GSS; Haiku reached 40% on WVS\. Sonnet and Llama\-70B remained at 0% invalid\. This is itself a practical result: a researcher relying on distribution prompts from a small model may silently lose most of their sample\.

The accuracy numbers elsewhere in the paper are*conditional on valid output*, which flatters the high\-invalid cells because invalid responses may not be missing at random\. We therefore report both bounds\. Where invalid rates are near zero \(all Claude and Llama\-70B cells\), the two coincide, so those results are unaffected\. Where they are high, treating each invalid output as a failed prediction, the appropriate accounting when a synthetic user must return a usable answer, widens the deficit sharply: Llama\-8B’s Style\-C accuracy falls from0\.510\.51to0\.220\.22on GSS and from0\.180\.18to0\.030\.03on WVS, and Haiku’s WVS Style\-C from0\.270\.27to0\.160\.16\. Because the non\-LLM baselines never emit invalid output, penalizing invalids can only widen the LLM’s gap to the baseline, never close it; the individual\-level conclusion is robust to either accounting\.

Third,predictions are unstable to prompt\-surface choices that should be irrelevant\. Prior work shows that survey\-style LLM answers are sensitive to option ordering and labeling\[[13](https://arxiv.org/html/2607.26348#bib.bib7)\]and that prompt architecture and persona formulation can themselves induce artifacts\[[7](https://arxiv.org/html/2607.26348#bib.bib14),[24](https://arxiv.org/html/2607.26348#bib.bib23)\]; we quantify this for our setting\. On the two clean closed models we measured the*prediction\-flip rate*\(the fraction of single\-answer predictions that change\) under two perturbations that leave the actual question and answer set unchanged: reversing the display order of the answer options, and reframing the identical demographics as a natural\-language persona sentence rather than a bulleted profile\. We benchmark these against a decoding\-noise floor: the flip rate between two decoding seeds of the*same*prompt\. On GSS, merely reversing option order flips9\.8%9\.8\\%\(Sonnet\) to10\.8%10\.8\\%\(Haiku\) of predictions, and the persona reframing flips a further∼9%\\sim\\\!9\\%, both roughly44–10×10\\timesthe seed\-noise floor \(1\.1%1\.1\\%and2\.6%2\.6\\%respectively\)\. An option\-order spot\-check on WVS shows the same effect more strongly \(14\.3%14\.3\\%flips for Sonnet,22\.6%22\.6\\%for Haiku\)\. Option order also moves aggregate fidelity \(e\.g\. Haiku GSS accuracy0\.587→0\.6060\.587\\\!\\rightarrow\\\!0\.606\), so a study’s conclusions can depend on an arbitrary presentation choice\. Together with the capability and format results, this means synthetic\-user validity is not only a model property but a prompt/interface property\.

### 4\.6RQ5: Both failures transfer across domains

The two headline failures \(no individual\-level advantage over the demographic baseline, and positive stereotyping\) appear in*both*the U\.S\. social\-attitude domain and the 63\-country values domain, for*all four*models and*both*families\. The individual\-level failure is in fact*stronger*in the cross\-cultural domain \(an 11–22 point deficit vs\. a rough tie\)\. This cross\-domain consistency is the basis for treating these as properties of current LLM synthetic\-user methods rather than artifacts of a single survey, model, or prompt\.

## 5Discussion

The two failures discussed in this paper stem from the same underlying cause: current LLMs, prompted with demographics, appear to answer by mapping a profile to a stereotyped “typical” response for that profile\. This is consistent with evidence that language models encode social associations\[[9](https://arxiv.org/html/2607.26348#bib.bib29),[4](https://arxiv.org/html/2607.26348#bib.bib33)\]and that persona conditioning steers models toward stereotyped abstractions rather than nuanced individuals\[[11](https://arxiv.org/html/2607.26348#bib.bib21)\]\. That mapping is too coarse to beat a demographic lookup at the individual level, and it is worse than the lookup when the answer space is large, as in the 10\-point WVS scales\. Yet it is also too tight: it treats demographics as more predictive of attitudes than they truly are\. The practical corollary is that a synthetic\-user study can look successful on an aggregate\-distribution metric, even one improved by distribution\-specialized training\[[10](https://arxiv.org/html/2607.26348#bib.bib5)\], while being invalid for any individual\- or subgroup\-level use, and while systematically exaggerating the demographic structure of opinion\. Because the aggregate metric is the one most often reported, this is a live risk for the field\.

#### Implications for decision support

These two failures map directly onto how synthetic users are used to support decisions\. A team that only checks aggregate fidelity, the common practice, can be misled in two ways at once\. Any decision keyed to*individuals or narrow segments*\(personalizing an offer, sizing a niche, or predicting how a specific demographic will react\) rests on individual\-level fidelity the models do not have; a demographic lookup would serve at least as well and often better, at a fraction of the cost\. And any decision that reads the*structure*of opinion off the simulation \(“this attitude splits sharply along political lines,” “this segment is monolithic”\) inherits the models’ over\-determination, which manufactures group differences that are far larger than they are in real people\. Our decision\-impact analysis \(Section[4\.4](https://arxiv.org/html/2607.26348#S4.SS4)\) puts numbers on this second failure: on the segment\-targeting task the models inflate between\-segment gaps two to fourfold, would send a team to the wrong segment in half of GSS and most WVS cases, and manufacture a nonexistent segment split in up to41%41\\%of value questions\. The danger is not random noise, which averages out, but this*structured*error, which propagates into the decision\. The framework we propose is meant to catch exactly this before the evidence is trusted: it turns “did the aggregate look right?” into a set of checks a team can run against held\-out human data to decide, for their specific question, whether synthetic\-user evidence is safe\.

#### A validation protocol

From these results we distill a checklist for researchers considering LLM synthetic users: \(1\) report individual\-level and aggregate\-level fidelity*separately*; \(2\) always benchmark individual fidelity against non\-LLM baselines computed on real held\-out data \(at least a demographic lookup, ideally a learned demographic model\) using distance\-aware or proper\-scoring metrics on ordinal scales; an LLM that does not beat them adds no individual\-level information; \(3\) report subgroup determinism \(e\.g\.Δ​η2\\Delta\\eta^\{2\}, or a coding\-invariant association measure\), not just group means, to detect stereotyping; \(4\) before acting on any segment\-level read, check the decision\-impact quantities \(gap inflation, wrong\-target rate, and spurious\-split rate\) against held\-out human data, since a distortion invisible in an aggregate metric can still flip a targeting decision; \(5\) report invalid/refusal rates, especially for distribution prompts and smaller models; \(6\) do not assume a larger or more capable model is a safer synthetic user; and \(7\) restrict validity claims to the population, domain, and level of analysis actually tested\.

#### Limitations

Our claims are scoped to the data we used: GSS respondents 2016–2024 and WVS Wave 7 across 63 countries, on the specific questions listed, with two prompt formats and two seeds, and to demographic prompting under survey\-simulation protocols, not to every elicitation strategy\. The LLM\-evaluated sample is about100100respondents per question, but the individual\-fidelity claims pool all rows in a domain \(n≈2,000n\\approx 2\{,\}000on GSS,n≈2,900n\\approx 2\{,\}900on WVS\) and every headline comparison carries a paired\-bootstrap95%95\\%interval, so the conclusions rest on those intervals rather than on any single question’s cell\. We do not claim these numbers characterize every model, prompt, or elicitation strategy; richer persona construction\[[20](https://arxiv.org/html/2607.26348#bib.bib22)\], few\-shot conditioning, or fine\-tuning could shift the individual\-fidelity and stereotyping numbers, and testing that is exactly the value of a fixed, reusable protocol\. We estimate human targets without survey weights; because applying the official GSS and WVS weights moves those targets by amounts one to three orders of magnitude smaller than the effects we report \(Appendix[F](https://arxiv.org/html/2607.26348#A6)\), this choice does not affect any claim\. Older public surveys such as these may appear in model pretraining data; we mitigate this by scoping GSS to recent waves and by noting that memorization would, if anything,*help*aggregate fidelity while leaving the individual\-level and over\-determination failures intact\. Finally, survey answers are stated opinions, not behavior; extending the protocol to behavioral tasks and to interactive multi\-agent simulations\[[28](https://arxiv.org/html/2607.26348#bib.bib18),[26](https://arxiv.org/html/2607.26348#bib.bib20)\]is future work\.

## 6Conclusion

Across two independent domains, four models, and two model families, and under the demographic\-prompting protocols we test, LLM synthetic users show two robust failures: they do not beat, and on cross\-cultural values fall well below, non\-LLM baselines at the individual level, and they systematically over\-determine demographics, treating identity as far more predictive of attitudes than it is among real people\. The individual\-level deficit survives distance\-aware and proper scoring, and the over\-determination survives a coding\-invariant association measure; model capability does not fix either failure\.

The contribution, however, is not merely the negative result that these systems fail\. It is a reusable evaluation framework for intelligent synthetic\-user systems: the cross\-domain benchmark, the suite of non\-LLM baselines scored with distance\-aware and proper\-scoring metrics, the stereotyping index and its coding\-invariant companion, and the validation protocol that ties them together, designed to be run before synthetic\-user evidence is deployed in a decision\-support workflow\. Rather than trusting an aggregate similarity number, a team can use this framework to determine, for their specific question, population, and level of analysis, whether an LLM synthetic user is fit for purpose or whether a trivial demographic predictor would serve at least as well\. As LLMs are increasingly wired into product, policy, and market decisions, that up\-front check is what turns a cautionary finding into an actionable safeguard\.

## Appendix AModel call settings

All four models \(Claude Haiku 4\.5, Claude Sonnet 4\.6, Llama\-3\.1\-8B\-Instruct, and Llama\-3\.3\-70B\-Instruct\) are called through a single common chat\-completion interface\. Decoding uses temperature1\.01\.0\(the model’s default sampling; where a model rejects an explicit temperature field it is omitted and the provider default applies\) and a token cap of100100\. Each \(model, prompt format, respondent, question\) cell is generated under two independent runs \(“seeds”0and11\); these differ only in sampling randomness and are used to estimate the run\-to\-run noise floor in RQ4\.

## Appendix BExact questions and answer scales

TablesLABEL:tab:gssqandLABEL:tab:wvsqgive every question verbatim, with its answer options and integer coding, whether it is treated as ordinal or nominal, and which metrics it enters\. GSS wordings and value labels are the standard GSS codebook text; WVS wordings and ordinal scales follow WorldValuesBench for comparability\. An item is ordinal when a mean answer is meaningful; only ordinal items enter theη2\\eta^\{2\}stereotyping index and the distance\-aware metrics \(MAE/EMD\), while all items enter accuracy, JS divergence, and the coding\-invariant Cramér’s V \(VV\)\. GSS prompt demographics are age, sex, race, highest degree, region, political views, and party identification \(subgroup axes: sex, race, degree, region, political views\); WVS prompt demographics are age group, sex, education level, settlement type, and country \(subgroup axes: country, education, age group, sex\)\.

Table 3:GSS questions: wording, answer options \(code=label\), type, and metrics entered\. “all”==acc, JS, MAE/EMD, log\-loss/Brier,η2\\eta^\{2\},VV\.VarQuestion wordingOptions \(code=label\)TypeMetricshappyTaken all together, how would you say things are these days – would you say that you are very happy, pretty happy, or not too happy?1=very happy; 2=pretty happy; 3=not too happyordinalalltrustGenerally speaking, would you say that most people can be trusted or that you can’t be too careful in dealing with people?1=most people can be trusted; 2=can’t be too careful; 3=dependsnominalacc, JS,VVfairDo you think most people would try to take advantage of you if they got a chance, or would they try to be fair?1=would take advantage of you; 2=would try to be fair; 3=dependsnominalacc, JS,VVhelpfulWould you say that most of the time people try to be helpful, or that they are mostly just looking out for themselves?1=try to be helpful; 2=looking out for themselves; 3=dependsnominalacc, JS,VVcappunDo you favor or oppose the death penalty for persons convicted of murder?1=favor; 2=opposenominalacc, JS,VVgrassDo you think the use of marijuana should be made legal or not?1=should be legal; 2=should not be legalnominalacc, JS,VVabanyPlease tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if the woman wants it for any reason\.1=yes; 2=nonominalacc, JS,VVgunlawWould you favor or oppose a law which would require a person to obtain a police permit before he or she could buy a gun?1=favor; 2=opposenominalacc, JS,VVfefamDo you agree or disagree with this statement: It is much better for everyone involved if the man is the achiever outside the home and the woman takes care of the home and family?1=strongly agree; 2=agree; 3=disagree; 4=strongly disagreeordinalallconfinanHow much confidence do you have in the people running banks and financial institutions – a great deal, only some, or hardly any?1=a great deal; 2=only some; 3=hardly anyordinalallTable 4:WVS questions \(WorldValuesBench probe set\): wording, scale, type, and metrics entered\. All are ordinal integer scales\.VarQuestion wordingScaleTypeMetricsQ48On a scale of 1 to 10, 1 meaning ‘None at all’ and 10 meaning ‘A great deal’, how much freedom of choice and control over your life do you feel you have?1–10ord\.allQ106On a scale of 1 to 10, 1 meaning ‘Incomes should be made more equal’ and 10 meaning ‘There should be greater incentives for individual effort’, where would you place your view?1–10ord\.allQ107On a scale of 1 to 10, 1 meaning ‘Private ownership of business should be increased’ and 10 meaning ‘Government ownership of business should be increased’, where would you place your view?1–10ord\.allQ108On a scale of 1 to 10, 1 meaning ‘The government should take more responsibility to ensure that everyone is provided for’ and 10 meaning ‘People should take more responsibility to provide for themselves’, where would you place your view?1–10ord\.allQ112On a scale of 1 to 10, 1 meaning ‘No corruption at all’ and 10 meaning ‘Abundant corruption’, how much corruption do you think there is in your country?1–10ord\.allQ113On a scale of 1 to 4, 1 meaning ‘None of them’ and 4 meaning ‘All of them’, how many state authorities do you think are involved in corruption?1–4ord\.allQ114On a scale of 1 to 4, 1 meaning ‘None of them’ and 4 meaning ‘All of them’, how many business executives do you think are involved in corruption?1–4ord\.allQ121On a scale of 1 to 5, 1 meaning ‘Very bad’ and 5 meaning ‘Very good’, what impact do you think immigrants have on the development of your country?1–5ord\.allQ122On a scale of 0 to 2, 0 meaning ‘Disagree’ and 2 meaning ‘Agree’, do you agree that immigration fills useful jobs in the labour market?0–2ord\.allQ123On a scale of 0 to 2, 0 meaning ‘Disagree’ and 2 meaning ‘Agree’, do you agree that immigration increases the crime rate?0–2ord\.allQ158On a scale of 1 to 10, 1 meaning ‘Completely disagree’ and 10 meaning ‘Completely agree’, how much do you agree that science and technology make our lives healthier, easier, and more comfortable?1–10ord\.allQ159On a scale of 1 to 10, 1 meaning ‘Completely disagree’ and 10 meaning ‘Completely agree’, how much do you agree that because of science and technology there will be more opportunities for the next generation?1–10ord\.allQ160On a scale of 1 to 10, 1 meaning ‘Completely disagree’ and 10 meaning ‘Completely agree’, how much do you agree that we depend too much on science and not enough on faith?1–10ord\.allQ164On a scale of 1 to 10, 1 meaning ‘Not at all important’ and 10 meaning ‘Very important’, how important is God in your life?1–10ord\.allQ177On a scale of 1 to 10, 1 meaning ‘Never justifiable’ and 10 meaning ‘Always justifiable’, how justifiable do you think it is to claim government benefits to which you are not entitled?1–10ord\.allQ178On a scale of 1 to 10, 1 meaning ‘Never justifiable’ and 10 meaning ‘Always justifiable’, how justifiable do you think it is to avoid paying a fare on public transport?1–10ord\.allHere “all” denotes accuracy, JS, MAE/EMD, log\-loss/Brier,η2\\eta^\{2\}, andVV, since every WVS item is ordinal\.

## Appendix CPrompt templates and demographic formatting

Demographics are rendered as a bulleted profile, one line per attribute, using human\-readable value labels \(e\.g\. “\- Political views \(liberal\-\-conservative\): conservative”\); for the persona perturbation \(RQ4\) the same attributes are rewritten as a single natural\-language sentence \(“*a person with …*”\)\. The two prompt formats share this demographic block and differ only in the response instruction:

*Single\-answer prompt \(Style A\)\.*

> You are simulating a single \{respondent noun\} with the following demographic profile: \{demographics\} Based only on this profile, predict how this specific person would answer the following survey question\. Question: \{question text\} Answer options: \{options\} Respond with only the single \{option letter\|\|number\} \(e\.g\. A\)\. No explanation\.

*Distribution prompt \(Style C\)\.*

> Consider a \{respondent noun\} with the following demographic profile: \{demographics\} Estimate the probability that this respondent would choose each answer option for the question below\. Probabilities must sum to 1\. Question: \{question text\} Answer options: \{options\} Return only valid JSON mapping each answer \{key\} to a probability, using exactly these keys: \{keys\}\. No other text\.

Answer options use short letters \(A, B, …\) when the options have text labels \(GSS\), and the scale number itself when options are numeric points on an ordinal scale \(WVS\), which avoids the eight\-letter limit on the longest scales\. In every case the prompt carries an explicit key→\\rightarrowanswer\-code mapping that the parser uses to recover the underlying code\.

## Appendix DParser rules and invalid\-output handling

Outputs are parsed deterministically\. For the single\-answer prompt with letter keys, the parser takes the first standalone capital letter that is a valid option key; with numeric keys it takes the first integer token that is a valid key, matching multi\-digit tokens so that “10” is not read as “1”\. For the distribution prompt, the parser extracts the first balanced JSON object, reads a probability for each declared key \(absent keys default to0\), and normalizes to sum to one; any response that is empty, contains no valid key, fails JSON parsing, or yields a non\-positive total is flagged*invalid/refused*\. Invalid outputs are excluded from the accuracy and distribution metrics \(a conditional\-on\-valid accounting\) and counted in the invalid\-rate stability metric \(RQ4\)\. Because invalids may not be missing at random, Section[4\.5](https://arxiv.org/html/2607.26348#S4.SS5)also reports the opposite bound \(each invalid counted as a failed prediction\), and the two accountings coincide wherever the invalid rate is near zero\. Prose\-wrapped answers are tolerated as long as a valid key is recoverable\.

## Appendix EBaseline construction and back\-off

All baselines are fit on the 50% “fit” fold \(respondents with an even hash of their id\) and evaluated on the evaluation\-sample rows, which are drawn from the “eval” fold, so no respondent is used for both\. The*question\-marginal*baseline predicts the single most frequent fit\-fold answer for the question\. The*demographic\-lookup*baseline estimates a conditional answer distribution for each demographic cell and predicts its mode, backing off when a cell has fewer than2020fit\-fold respondents: for GSS the back\-off is \{degree, race, sex, political views\}→\\rightarrow\{degree, race\}→\\rightarrowquestion marginal; for WVS it is \{country, education, age group\}→\\rightarrow\{country\}→\\rightarrowquestion marginal, and we report the \{country\} level separately as a country\-only baseline\. The*logistic*baseline is a multinomial logistic regression on the one\-hot\-encoded prompt demographics, fit per question \(≤2000\\leq\\\!2000iterations\); the*random\-forest*baseline uses300300trees on the same features\. The lookup and logistic baselines also emit a full predicted distribution, used as the reference for the log\-loss and Brier comparisons\.

## Appendix FEstimation robustness

This appendix records the four estimation checks summarized in Section[3\.6](https://arxiv.org/html/2607.26348#S3.SS6); all are computed on the same cached outputs as the main results, with no additional model calls\.

*Multiple comparisons \(FDR\)\.*For each \(question, axis\) pair entering theη2\\eta^\{2\}stereotyping index we compute a one\-sided bootstrappp\-value forH0:Δ​η2≤0H\_\{0\}\\\!:\\Delta\\eta^\{2\}\\leq 0and apply a Benjamini–Hochberg correction atq=0\.05q=0\.05across all pairs within a model and prompt style\. The number of significant, positively\-signed pairs is nearly identical before and after correction, because the effects are large: for the single\-answer prompt, Sonnet is32/3232/32\(WVS\) and10/1210/12\(GSS\) both before and after correction; Haiku31/32→31/3231/32\\to 31/32\(WVS\); Llama\-70B31/32→31/3231/32\\to 31/32\(WVS\) and7/8→7/87/8\\to 7/8\(GSS\)\. Over\-determination is therefore not an artifact of uncorrected multiple testing\.

*Respondent\-clustered bootstrap\.*The individual\-fidelity sample contains a small share of respondents who answer more than one question \(7\.6%7\.6\\%on GSS,1\.8%1\.8\\%on WVS\)\. Recomputing the RQ1 accuracy margin over the baseline with a cluster bootstrap \(resampling respondents, then taking all of their rows\) leaves the95%95\\%intervals essentially unchanged from the row\-level intervals: on WVS every margin still excludes zero \(e\.g\. Sonnet single\-answer−0\.152\-0\.152,\[−0\.180,−0\.121\]\[\-0\.180,\-0\.121\]; Haiku−0\.218\-0\.218,\[−0\.248,−0\.187\]\[\-0\.248,\-0\.187\]\), and on GSS the Llama\-8B deficits remain significant while the Claude cells remain ties, exactly as in the row\-level analysis\. Within\-respondent correlation does not drive the result\.

*Survey\-weight robustness\.*GSS and WVS provide official weights \(wtssps, coverage100%100\\%;W\_WEIGHT, coverage100%100\\%on the in\-scope pool\)\. Applying them to the two human\-side quantities the claims depend on changes them negligibly: the human population answer distribution moves by a mean Jensen–Shannon divergence of0\.00010\.0001\(GSS\) and<0\.0001<\\\!0\.0001\(WVS\), with a maximum over questions of0\.00030\.0003; the humanη2\\eta^\{2\}that the stereotyping index subtracts moves by a mean absolute0\.00160\.0016\(GSS\) and0\.00120\.0012\(WVS\), with a maximum of0\.00650\.0065\. These are one to three orders of magnitude below the effects we report, so the unweighted analysis is adequate for every claim in the paper\.

*Sample adequacy\.*Individual\-fidelity margins pool all evaluation rows per domain \(n=1,986n=1\{,\}986scored \(respondent, question\) pairs on GSS,n=2,916n=2\{,\}916on WVS for the clean cells\) rather than the≈100\\approx\\\!100respondents of any single question, and every headline comparison carries a paired\-bootstrap95%95\\%interval computed on those rows; the intervals, not a nominal per\-question count, are what license the claims\.

## References

- \[1\]\(2023\)Using large language models to simulate multiple humans and replicate human subject studies\.InInternational conference on machine learning,pp\. 337–371\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[2\]L\. P\. Argyle, E\. C\. Busby, N\. Fulda, J\. R\. Gubler, C\. Rytting, and D\. Wingate\(2023\)Out of one, many: using language models to simulate human samples\.Political Analysis31\(3\),pp\. 337–351\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[3\]D\. Arnott and G\. Pervan\(2005\)A critical analysis of decision support systems research\.Journal of information technology20\(2\),pp\. 67–87\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[4\]E\. M\. Bender, T\. Gebru, A\. McMillan\-Major, and S\. Shmitchell\(2021\)On the dangers of stochastic parrots: can language models be too big?\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 610–623\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.p1.1)\.
- \[5\]J\. Bisbee, J\. D\. Clinton, C\. Dorff, B\. Kenkel, and J\. M\. Larson\(2024\)Synthetic replacements for human survey data? the perils of large language models\.Political Analysis32\(4\),pp\. 401–416\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[6\]T\. Bolukbasi, K\. Chang, J\. Y\. Zou, V\. Saligrama, and A\. T\. Kalai\(2016\)Man is to computer programmer as woman is to homemaker? debiasing word embeddings\.Advances in neural information processing systems29\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1)\.
- \[7\]M\. Brucks and O\. Toubia\(2025\)Prompt architecture induces methodological artifacts in large language models\.PloS one20\(4\),pp\. e0319159\.Cited by:[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1),[§4\.5](https://arxiv.org/html/2607.26348#S4.SS5.p3.10)\.
- \[8\]J\. Buolamwini and T\. Gebru\(2018\)Gender shades: intersectional accuracy disparities in commercial gender classification\.InConference on fairness, accountability and transparency,pp\. 77–91\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1),[§3\.5](https://arxiv.org/html/2607.26348#S3.SS5.p5.4)\.
- \[9\]A\. Caliskan, J\. J\. Bryson, and A\. Narayanan\(2017\)Semantics derived automatically from language corpora contain human\-like biases\.Science356\(6334\),pp\. 183–186\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.p1.1)\.
- \[10\]Y\. Cao, H\. Liu, A\. Arora, I\. Augenstein, P\. Röttger, and D\. Hershcovich\(2025\)Specializing large language models to simulate survey response distributions for global populations\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 3141–3154\.Cited by:[§2\.2](https://arxiv.org/html/2607.26348#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.p1.1)\.
- \[11\]A\. Deshpande, V\. Murahari, T\. Rajpurohit, A\. Kalyan, and K\. Narasimhan\(2023\)Toxicity in chatgpt: analyzing persona\-assigned language models\.InFindings of the association for computational linguistics: EMNLP 2023,pp\. 1236–1270\.Cited by:[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.p1.1)\.
- \[12\]D\. Dillion, N\. Tandon, Y\. Gu, and K\. Gray\(2023\)Can ai language models replace human participants?\.Trends in Cognitive Sciences27\(7\),pp\. 597–600\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[13\]R\. Dominguez\-Olmedo, M\. Hardt, and C\. Mendler\-Dünner\(2024\)Questioning the survey responses of large language models\.Advances in Neural Information Processing Systems37,pp\. 45850–45878\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1),[§4\.5](https://arxiv.org/html/2607.26348#S4.SS5.p3.10)\.
- \[14\]Y\. Gao, D\. Lee, G\. Burtch, and S\. Fazelpour\(2025\)Take caution in using llms as human surrogates\.Proceedings of the National Academy of Sciences122\(24\),pp\. e2501660122\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[15\]T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. D\. Iii, and K\. Crawford\(2021\)Datasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[16\]R\. M\. Groves and L\. Lyberg\(2010\)Total survey error: past, present, and future\.Public opinion quarterly74\(5\),pp\. 849–879\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1),[§3\.5](https://arxiv.org/html/2607.26348#S3.SS5.p1.1)\.
- \[17\]P\. Hämäläinen, M\. Tavast, and A\. Kunnari\(2023\)Evaluating large language models in generating synthetic hci research data: a case study\.InProceedings of the 2023 CHI conference on human factors in computing systems,pp\. 1–19\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[18\]M\. Hardt, E\. Price, and N\. Srebro\(2016\)Equality of opportunity in supervised learning\.Advances in neural information processing systems29\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1),[§3\.5](https://arxiv.org/html/2607.26348#S3.SS5.p5.4)\.
- \[19\]J\. J\. Horton, A\. Filippas, and B\. S\. Manning\(2023\)Large language models as simulated economic agents: what can we learn from homo silicus?\.Technical reportNational Bureau of Economic Research\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[20\]L\. Li, H\. Chen, H\. Namkoong, and T\. Peng\(2026\)Llm generated persona is a promise with a catch\.Advances in Neural Information Processing Systems38\.Cited by:[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.SS0.SSS0.Px3.p1.4)\.
- \[21\]P\. Li, N\. Castelo, Z\. Katona, and M\. Sarvary\(2024\)Frontiers: determining the validity of large language models for automated perceptual analysis\.Marketing Science43\(2\),pp\. 254–266\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1)\.
- \[22\]P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research2023\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1),[§3\.5](https://arxiv.org/html/2607.26348#S3.SS5.p1.1)\.
- \[23\]Z\. Lin\(2025\)Six fallacies in substituting large language models for human participants\.Advances in Methods and Practices in Psychological Science8\(3\),pp\. 25152459251357566\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[24\]M\. Lutz, I\. Sen, G\. Ahnert, E\. Rogers, and M\. Strohmaier\(2025\)The prompt makes the person \(a\): a systematic evaluation of sociodemographic persona prompting for large language models\.arXiv preprint arXiv:2507\.16076\.Cited by:[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1),[§4\.5](https://arxiv.org/html/2607.26348#S4.SS5.p3.10)\.
- \[25\]M\. Mitchell, S\. Wu, A\. Zaldivar, P\. Barnes, L\. Vasserman, B\. Hutchinson, E\. Spitzer, I\. D\. Raji, and T\. Gebru\(2019\)Model cards for model reporting\.InProceedings of the conference on fairness, accountability, and transparency,pp\. 220–229\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[26\]X\. Mou, X\. Ding, Q\. He, L\. Wang, J\. Liang, X\. Zhang, L\. Sun, J\. Lin, J\. Zhou, H\. Xuanjing,et al\.\(2026\)From individual to society: a survey on social simulation driven by large language model\-based agents\.ACM Computing Surveys58\(11\),pp\. 1–41\.Cited by:[§2\.4](https://arxiv.org/html/2607.26348#S2.SS4.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.SS0.SSS0.Px3.p1.4)\.
- \[27\]R\. M\. O’Keefe and D\. E\. O’Leary\(1993\)Expert system verification and validation: a survey and tutorial\.Artificial intelligence review7\(1\),pp\. 3–42\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p2.1),[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[28\]J\. S\. Park, J\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein\(2023\)Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th annual acm symposium on user interface software and technology,pp\. 1–22\.Cited by:[§2\.4](https://arxiv.org/html/2607.26348#S2.SS4.p1.1),[§5](https://arxiv.org/html/2607.26348#S5.SS0.SSS0.Px3.p1.4)\.
- \[29\]J\. S\. Park, L\. Popowski, C\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein\(2022\)Social simulacra: creating populated prototypes for social computing systems\.InProceedings of the 35th annual ACM symposium on user interface software and technology,pp\. 1–18\.Cited by:[§2\.4](https://arxiv.org/html/2607.26348#S2.SS4.p1.1)\.
- \[30\]A\. Parrish, A\. Chen, N\. Nangia, V\. Padmakumar, J\. Phang, J\. Thompson, P\. M\. Htut, and S\. R\. Bowman\(2022\)BBQ: a hand\-built bias benchmark for question answering\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2086–2105\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1)\.
- \[31\]M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh\(2020\)Beyond accuracy: behavioral testing of nlp models with checklist\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 4902–4912\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1),[§3\.5](https://arxiv.org/html/2607.26348#S3.SS5.p1.1)\.
- \[32\]A\. Saibene, M\. Assale, and M\. Giltri\(2021\)Expert systems: definitions, advantages and issues in medical field applications\.Expert Systems with Applications177,pp\. 114900\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[33\]S\. Santurkar, E\. Durmus, F\. Ladhak, C\. Lee, P\. Liang, and T\. Hashimoto\(2023\)Whose opinions do language models reflect?\.InInternational conference on machine learning,pp\. 29971–30004\.Cited by:[§2\.1](https://arxiv.org/html/2607.26348#S2.SS1.p1.1)\.
- \[34\]M\. Sarstedt, S\. J\. Adler, L\. Rau, and B\. Schmitt\(2024\)Using large language models to generate silicon samples in consumer and marketing research: challenges, opportunities, and guidelines\.Psychology & Marketing41\(6\),pp\. 1254–1270\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.26348#S2.SS3.p1.1)\.
- \[35\]A\. D\. Selbst, D\. Boyd, S\. A\. Friedler, S\. Venkatasubramanian, and J\. Vertesi\(2019\)Fairness and abstraction in sociotechnical systems\.InProceedings of the conference on fairness, accountability, and transparency,pp\. 59–68\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1)\.
- \[36\]E\. Sheng, K\. Chang, P\. Natarajan, and N\. Peng\(2019\)The woman worked as a babysitter: on biases in language generation\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 3407–3412\.Cited by:[§2\.5](https://arxiv.org/html/2607.26348#S2.SS5.p1.1)\.
- \[37\]G\. Shmueli and O\. R\. Koppius\(2011\)Predictive analytics in information systems research1\.MIS quarterly35\(3\),pp\. 553–572\.Cited by:[§1](https://arxiv.org/html/2607.26348#S1.p2.1),[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1),[§2\.7](https://arxiv.org/html/2607.26348#S2.SS7.p1.1),[§3\.4](https://arxiv.org/html/2607.26348#S3.SS4.p1.2)\.
- \[38\]A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.\(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on machine learning research\.Cited by:[§2\.6](https://arxiv.org/html/2607.26348#S2.SS6.p1.1)\.
- \[39\]J\. Wang, Z\. Zhao, T\. Ni, and Z\. Wei\(2025\)SocioBench: modeling human behavior in sociological surveys with large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 26268–26300\.Cited by:[§2\.2](https://arxiv.org/html/2607.26348#S2.SS2.p1.1)\.
- \[40\]W\. Zhao, D\. Mondal, N\. Tandon, D\. Dillion, K\. Gray, and Y\. Gu\(2024\)Worldvaluesbench: a large\-scale benchmark dataset for multi\-cultural value awareness of language models\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 17696–17706\.Cited by:[§2\.2](https://arxiv.org/html/2607.26348#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2607.26348#S3.SS1.p3.1)\.
- \[41\]C\. Ziems, W\. Held, O\. Shaikh, J\. Chen, Z\. Zhang, and D\. Yang\(2024\)Can large language models transform computational social science?\.Computational Linguistics50\(1\),pp\. 237–291\.Cited by:[§2\.4](https://arxiv.org/html/2607.26348#S2.SS4.p1.1)\.