基于LLM的人类行为模拟可靠性解析

# Understanding Reliability in LLM-based Human Behavior Simulation
Source: [https://arxiv.org/html/2609.25066](https://arxiv.org/html/2609.25066)
Lei WangYuanzi LiXu Chen††thanks:Corresponding author\.Affiliation:Gaoling School of Artificial Intelligence, Renmin University of ChinaAffiliation:\{wang\_pei, xu\.chen\}@ruc\.edu\.cn

###### Abstract

Large language models \(LLMs\) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead social science conclusions\. However, existing evaluations focus on end\-to\-end scores, leaving it unclear how different aspects of the simulation process interact to determine reliability\. We propose ReliMap, which decomposes LLM\-based human behavior simulation into three structured layers and evaluates reliability at both the individual level \(R1\) and population level \(R2\) across three configuration dimensions: model capacity, profile completeness, and population coverage\. Through experiments across four simulation tasks and eleven LLMs, we find that all models exhibit substantial distributional bias without profile conditioning\. Profile conditioning reduces this bias with diminishing returns\. Larger models benefit more, and attribute informativeness matters more than quantity\. Critically, R1 gains do not reliably transfer to R2—individual and population\-level reliability can move in opposite directions\. At the population layer, increasing coverage reduces variance but not systematic bias, with R2 stabilizing at around 50–100 individuals\. These findings highlight that reliable simulation cannot be achieved by optimizing any single layer in isolation, but requires coordinated improvement across all three\.111Our project page is available at[https://yupei\-wang\.github\.io/understand\_reliability/](https://yupei-wang.github.io/understand_reliability/)\.

## 1Introduction

Human experiments and surveys remain the gold standard for studying behavioral tendencies and social attitudes, yet their scale is fundamentally constrained by cost and participant availability[Aher et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib1);[Argyle et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib2);[Bisbee et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib5)\. Large language models \(LLMs\) have emerged as a promising alternative, enabling rapid, low\-cost simulation of human responses at scale\. This has sparked growing interest in using LLMs as computational proxies for human participants[Dominguez\-Olmedo et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib12);[Hu et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib19);[Manning et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib23);[Binz et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib4);[Hu et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib18)\.

![Refer to caption](https://arxiv.org/html/2609.25066v1/figures/intro.png)Figure 1:LLM\-based Human Behavior Simulation As a Hierarchical Structure\.![Refer to caption](https://arxiv.org/html/2609.25066v1/figures/main_v8.png)Figure 2:ReliMap Overview: Three\-Layer Structure with Configurable Dimensions and Two\-Level Reliability Evaluation\.Recent years have seen growing efforts in this direction, spanning individual\-level response prediction[Aher et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib1);[Hu et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib18);[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35)and population\-level distribution matching[Huang et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib20);[Wang et al\. \(2025b\)](https://arxiv.org/html/2609.25066#bib.bib34)\. Yet despite this breadth, the field remains fragmented\. On the methodology side, some work focuses on prompting strategies and persona engineering to improve per\-respondent simulation accuracy[Bisbee et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib5);[Cho et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib8);[Moon et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib26), while others fine\-tune or align models to better reproduce empirical responses[Chu et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib9);[Suh et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib31);[Kolluri et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib22)\. On the evaluation side, some work assesses whether simulated individual responses match ground\-truth labels[Hewitt et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib17), while others examine whether simulated population distributions align with human survey distributions[Huang et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib20);[Wang et al\. \(2025b\)](https://arxiv.org/html/2609.25066#bib.bib34)\. These lines of work each operate within a narrow scope, addressing one aspect of simulation in isolation\. Consequently, it remains unclear how different configuration choices interact to drive reliability\. Researchers lack the systematic understanding needed to diagnose and improve simulation quality\.

We argue that this gap stems from treating simulation as a single\-step process, when in fact it is hierarchical\. Our key insight is that this hierarchical process unfolds in three layers, each building upon the previous one\. As illustrated in Figure[1](https://arxiv.org/html/2609.25066#S1.F1), these three layers are: \(1\) Base distribution: The model induces an intrinsic behavioral distribution shaped by pretraining and alignment, capturing its default tendency before any individual information is introduced\. \(2\) Profile conditioning: This base distribution is then conditioned on individual profile information—such as demographics, values, or prior experiences—shifting behavioral tendencies toward those of specific personas\. \(3\) Population aggregation: These individualized distributions are finally aggregated across a sampled population, where coverage and sampling strategy determine how faithfully the simulation reflects the target population\. Building on this structure, we introduceReliMapto make reliability analysis more structured and interpretable\. Specifically, we systematically examine three key configuration dimensions—model capacity, profile completeness, and population coverage—and evaluate their joint effects at both the individual level \(R1\) and the population level \(R2\)\.

Guided by ReliMap, we conduct experiments on four simulation tasks with eleven LLMs\. All models exhibit substantial population\-level distributional bias without profile conditioning, with TVD varying widely across models and tasks\. Our analysis reveals six findings: \(1\) Richer profiles improve R1 with diminishing returns\. \(2\) Profile scaling behavior is strongly model\-dependent: larger models benefit more from additional profile information\. \(3\) Predictive information is highly concentrated in a small subset of attributes\. \(4\) Attribute informativeness matters more than attribute count\. \(5\) R1 gains do not reliably transfer to R2: individual accuracy improvements can coincide with worsening population\-level alignment, revealing a fundamental decoupling between the two levels\. \(6\) Population coverage reduces variance but not systematic bias, with R2 stabilizing at around 50–100 individuals\. Together, these findings deepen our understanding of what drives reliability in LLM\-based human behavior simulation\.

In summary, our contributions are threefold: \(1\) We introduce ReliMap, which transforms reliability from a single score into a decomposable analysis across three layers, three configuration dimensions, and two evaluation levels, providing a principled framework for understanding and diagnosing LLM\-based human behavior simulation\. \(2\) We conduct systematic experiments across four simulation tasks and eleven LLMs, providing a comprehensive empirical examination of how configuration choices shape reliability at each simulation layer\. \(3\) We reveal six findings that fundamentally advance our understanding of simulation reliability, exposing the limitations of current approaches and offering new insights into what it takes to achieve reliable LLM\-based human behavior simulation\.

## 2ReliMap

### 2\.1Hierarchical Structure

As illustrated in Figure[2](https://arxiv.org/html/2609.25066#S1.F2)\(a\), we characterize LLM\-based human behavior simulation as a three\-layer generative process, where each layer corresponds to a natural stage of how simulation is produced\.

#### Layer I: Base distribution\.

The first layer captures the model’s default responses without any personal profile information\. It is the baseline distribution induced under a given scenario\. This distribution is mainly shaped by the model architecture, pretraining data, and alignment strategy\. Formally, letfψf\_\{\\psi\}denote a pretrained LLM with parametersψ\\psi\. Given a scenarioss, we define the base distribution as:

ℙs≜Pfψ​\(y∣s\)\.\\mathbb\{P\}\_\{s\}\\;\\triangleq\\;P\_\{f\_\{\\psi\}\}\(y\\mid s\)\.\(1\)
Figure 3:Effect of profile completeness on population simulation reliability\.Table 1:TVD↓\\downarrowbetween LLM\-induced base response distributions and human survey distributions \(Layer I analysis; motivates the profile\-conditioning study of Findings 1–5\)\.
#### Layer II: Profile conditioning\.

Building on the baseline distribution, the model is provided with an individual profile, such as geographic information, psychological tendencies, and past behaviors, yielding a profile\-conditioned distribution\. Formally, given an individual profilexx, we define the profile\-conditional outcome distribution as

ℙs,x≜Pfψ​\(y∣s,x\)\.\\mathbb\{P\}\_\{s,x\}\\;\\triangleq\\;P\_\{f\_\{\\psi\}\}\(y\\mid s,x\)\.\(2\)
\(a\)\(a\)cc\-Acc curves across model families\(b\)\(b\)cc\-ACC curves across model scales\(c\)\(c\)cc\-ACC curves across reasoning variants
Figure 4:Changes in individual\-level simulation accuracy as profile completenessccincreases across models\. The x\-axisccdenotes the profile attribute coverage rate, i\.e\., the percentage of attributes provided to the model out of all available attributes in the dataset\. At each level of c, attributes are sampled randomly\.
#### Layer III: Population aggregation\.

To obtain population\-level simulation results, the third layer aggregates the simulated outputs of sampled individuals to form a population distribution\. Formally, we consider a finite population panel of N individuals𝒳N=\{xi\}i=1N\\mathcal\{X\}\_\{N\}=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}\. Aggregating individualized distributions of𝒳N\\mathcal\{X\}\_\{N\}yields the population\-aggregated outcome distribution:

ℙs,𝒳N≜1N​∑i=1NPfψ​\(y∣s,xi\)\.\\mathbb\{P\}\_\{s,\\mathcal\{X\}\_\{N\}\}\\;\\triangleq\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}P\_\{f\_\{\\psi\}\}\(y\\mid s,x\_\{i\}\)\.\(3\)Through the sequential generation and propagation from Layer I to Layer III, a standard LLM\-based human behavior simulation can be completed\.

### 2\.2Configuration Dimensions

We focus on three configuration dimensions that collectively capture the most critical controllable factors in the simulation setup, rather than implementation\-level choices such as prompt templates or decoding strategies\. As shown in Figure[2](https://arxiv.org/html/2609.25066#S1.F2)\(b\), three configuration dimensions are:

#### Model capacity \(ℳ\\mathcal\{M\}\)\.

It concerns the choice of the base model, such as the model architecture and model size\. This choice affects the simulation from Layer I and continues to propagate to later layers\.

#### Profile completeness \(cc\)\.

It captures how complete the available user information is\. In principle, richer profiles should enable more accurate simulation of individual behavior\. Yet in LLM\-based simulation, the effect need not be linear\. We therefore study how profile completeness shapes reliability, including when it matters and how strongly\. It affects Layer II and propagates to Layer III\.222We quantifyccas the proportion of attributes provided in a given profile relative to the total number of available attributes\.

#### Population coverage \(NN\)\.

It refers to the coverage of the sample used in population simulation\. Intuitively, broader coverage should lead to more reliable population\-level results\. However, this relationship needs to be quantified\. We ask how much coverage is sufficient for a reliable population simulation, and how much additional gain we get from increasing coverage further\. It affects Layer III\.

### 2\.3Two\-Level Reliability Evaluation

As shown in Figure[2](https://arxiv.org/html/2609.25066#S1.F2)\(c\), we evaluate simulation reliability at two complementary levels: individual\-level reliability \(R1\), which measures per\-respondent prediction accuracy, and population\-level reliability \(R2\), which measures how faithfully the aggregated simulation reproduces the true population distribution\. These two levels capture distinct aspects of simulation quality and are not interchangeable\. Intuitively, stronger R1 should translate into stronger R2, yet this coupling is not guaranteed\. We therefore treat their joint evaluation as a necessary condition for a complete assessment of simulation reliability\.

We state the R2 evaluation target precisely: R2 measures whether the model can reconstruct the response/preference distribution of a*specified target group*\. In our experiments, that target group is the retained analytical population after filtering respondents with complete and valid profile fields \(see Appendix[C](https://arxiv.org/html/2609.25066#A3)\), rather than the full original survey population, and we do not apply survey weights\.

#### Theoretical Insight

The three\-layer structure, configuration dimensions, and two\-level evaluation together form ReliMap\. A key theoretical insight underlies this framework: population\-level mismatch arises from multiple independent sources across simulation layers, and reducing one source does not automatically compensate for the others\. Full derivations are provided in Appendix[B](https://arxiv.org/html/2609.25066#A2)\.

## 3Experiments

### 3\.1Setup

#### Data\.

We conduct experiments on four datasets\. The first three are established survey datasets: partisan preference \(Party\) from the European Social Survey[ESS ERIC \(2025\)](https://arxiv.org/html/2609.25066#bib.bib14), immigration attitude \(Immigration\) from the World Values Survey[Association \(2022\)](https://arxiv.org/html/2609.25066#bib.bib3), and religious stance \(Religion\) from SocioBench[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35)\. The fourth, Media, differs from the others in two ways\. It is a Chinese\-language dataset, and it captures a real\-world dynamic event—public attitude toward the Luckin Coffee stock collapse—rather than a structured survey\. We collect it from a public social media corpus\.333Raw data sourced from[https://giser2000\.github\.io/geodata\.github\.io/\#download](https://giser2000.github.io/geodata.github.io/#download)\.More details are in Appendix[C](https://arxiv.org/html/2609.25066#A3)\.

#### Metric\.

We evaluate simulation reliability at two levels\. For R1, we use accuracy \(ACC\), which directly measures whether the model’s predicted response matches the ground\-truth individual behavior\. For R2, we use total variation distance \(TVD\), which quantifies the gap between the simulated and ground\-truth population distributions\. We choose ACC and TVD because they directly match our discrete\-choice R1/R2 setup and are widely adopted in prior work; we additionally verify that the main R2 trends remain consistent under KL divergence, Jensen–Shannon divergence \(JSD\), and Earth Mover’s Distance \(EMD\), as reported in Appendix[G](https://arxiv.org/html/2609.25066#A7)\.

#### Model\.

We evaluate eleven LLMs covering a diverse range of model families, scales, and reasoning paradigms\. Proprietary models include GPT\-4\.1[OpenAI \(2025\)](https://arxiv.org/html/2609.25066#bib.bib27)and Gemini\-2\.5\-Pro[Comanici et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib10)\. Open\-source models include DeepSeek\-V3\.2\-Exp[DeepSeek \(2025\)](https://arxiv.org/html/2609.25066#bib.bib11)and DeepSeek\-R1[Guo et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib16), as well as seven models from the Qwen3 family[Yang et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib37)spanning different parameter scales and instruct/thinking variants\. This diversity ensures that our findings are not specific to any single model family or architecture\.

### 3\.2Layer I — Base distribution

In Layer I, we examine the model’s base behavioral tendency without profile conditioning\. We provide no profile information and only input the task context, making one prediction for each instance in the full dataset\. Table[1](https://arxiv.org/html/2609.25066#S2.T1)shows the results\.

TVD scores exhibit substantial variation across models and tasks\. Even within a single task, models can differ markedly in distributional error—for example, on Party, the TVD of Qwen3\-14B and Qwen3\-235B\-A22B\-Instruct differs by approximately 0\.45\. Moreover, performance shows no consistent correlation with model scale or general capacity, suggesting that population\-level distribution alignment constitutes a distinct capacity dimension\. Crucially, all models exhibit non\-trivial TVD across all tasks, indicating that profile conditioning is necessary to avoid systematic distributional bias\.

### 3\.3Layer II — Profile Conditioning

We investigate the impact ofModel CapacityandProfile Completeness444Unless otherwise specified, attributes at each level ofccare randomly sampled by default\.on R1, and obtained the following findings\.

#### Finding 1: More Profile Context Helps, but Saturates Gradually

We progressively increase the number of profile attributes and observe the corresponding changes in R1\. The results are shown in Figure[4](https://arxiv.org/html/2609.25066#S2.F4)\. We find that across four tasks, ACC increases with the number of attributes, but with diminishing marginal gains\. This aligns with our intuition that richer user profiles provide models with more personal characteristics, thereby facilitating behavioral simulation\. The diminishing returns can be attributed to two factors\. On the one hand, newly added attributes may be uninformative or redundant relative to existing ones\. On the other hand, processing denser profiles requires stronger contextual understanding and reasoning from the model\.

#### Finding 2: Profile Scaling Behavior is Strongly Model\-Dependent

To test whether the diminishing returns reflect a capacity bottleneck, we compare how different models scale with profile completeness\.

Model Families\.As shown in Figure[4\(a\)](https://arxiv.org/html/2609.25066#S2.F4.sf1), we find thatperformance trends at higherccare model\-dependent\. Asccgrows large, some models’ accuracy converges to a similar level and continues to increase such as Gemini\-2\.5\-Pro, DeepSeek\-V3\.2\-Exp, GPT\-4\.1 and Qwen3\-235B\-A22B\-Instruct forParty, while others plateau at a lower level such as Qwen3\-8B and Qwen3\-32B, creating a clear performance split\. GPT\-4\.1 and DeepSeek\-V3\.2\-Exp consistently perform best across all three tasks; asccincreases, their ACC continues to improve substantially, indicating strong potential for further gains\.

Model Size\.As shown in Figure[4\(b\)](https://arxiv.org/html/2609.25066#S2.F4.sf2), we observe a broadly consistent trend across the three tasks:at the same profile completenesscc, larger models tend to achieve higher accuracy and continue to improve asccincreases\.Within the Qwen3 family, Qwen3\-32B consistently outperforms Qwen3\-14B; under the MoE setting, Qwen3\-235B\-A22B\-Instruct also generally outperforms Qwen3\-30B\-A3B\-Instruct\.

Instruct and Thinking variants\.Figure[4\(c\)](https://arxiv.org/html/2609.25066#S2.F4.sf3)compares Instruct and Thinking models\. We find no consistent evidence that Thinking models outperform Instruct models\. Although Thinking models have been shown to demonstrate superior reasoning on mathematical and logical tasks, their reasoning abilities may differ from those needed for behavioral simulation\. Simulating human behavior requires understanding social contexts, individual traits, and behavioral patterns that are potentially distinct from formal reasoning skills\.

#### Finding 3: Behavioral Information Is Highly Concentrated Among Few Attributes

Table 2:Attribute informativeness statistics supporting Finding 3\. Gini measures ASPG concentration across attributes \(0 = uniform, 1 = maximally concentrated\)\. Top 20% info and Bot 50% info denote the cumulative predictive information contributed by the top 20% and bottom 50% of attributes, respectively\. Info@c=50% reports cumulative information captured at half profile completeness under random \(rand\.\) and informativeness\-weighted \(wtd\.\) sampling\.To examine why performance saturates as profile completeness increases, we analyze the predictive information carried by individual attributes\. We defineAttribute\-Specific Predictive Gain \(ASPG\)as

ASPG​\(a\)=I⁡\(a,Y\)H⁡\(Y\),\\text\{ASPG\}\(a\)=\\frac\{I\(a;Y\)\}\{H\(Y\)\},\(4\)whereI⁡\(a,Y\)I\(a;Y\)is the mutual information between attributeaaand target labelYY, andH⁡\(Y\)H\(Y\)is the label entropy\. ASPG measures how much uncertainty about the target behavior can be reduced by observing a single attribute\.555ASPG analysis excludes the Media task, which uses unstructured historical posts rather than predefined attributes\.

As shown in Table[2](https://arxiv.org/html/2609.25066#S3.T2), ASPG distributions are strongly right\-skewed across all tasks\. The top 20% of attributes contribute roughly half of the total predictive information, while the bottom 50% contribute less than 22%\. Under the same completeness budget \(c=50%c=50\\%\), informativeness\-weighted sampling captures 79–86% of total information, compared to only 44–47% under random sampling\. These results show that predictive information is unevenly distributed across attributes, highlighting the importance of attribute selection\.

#### Finding 4: Attribute Informativeness Matters More Than Attribute Count

Table 3:Pearson correlations between accuracy and two profile completeness metrics, supporting Finding 4\.rQr\_\{Q\}: quantity\-based completeness;rIr\_\{I\}: information\-based completeness\.Δ=rI−rQ\\Delta=r\_\{I\}\-r\_\{Q\}\.Table[3](https://arxiv.org/html/2609.25066#S3.T3)compares two ways of measuring the correlation between profile completeness and ACC:rQr\_\{Q\}measures the Pearson correlation between the proportion of provided attributes and ACC, andrIr\_\{I\}measures the Pearson correlation between the cumulative ASPG of provided attributes and ACC\. Across most models and tasks,rIr\_\{I\}consistently exceedsrQr\_\{Q\}\. On average, information\-based completeness improves the correlation with accuracy by\+0\.1449onPartyand\+0\.1752onImmigration, indicating that the predictive information carried by attributes matters more than their count\.

The trend is particularly strong for larger models such as GPT\-4\.1 and DeepSeek\-V3\.2\-Exp, which achieve near\-perfect correlations betweenrIr\_\{I\}and accuracy on several tasks\. In contrast, the improvement is weaker or negative onReligion, suggesting that the usefulness of informative attributes depends on task characteristics\. Although ASPG relies on ground\-truth labels and may not be directly applicable in real\-world deployment, the results demonstrate that attribute quality is generally more important than attribute quantity\.

We note that ASPG is a*single\-attribute*statistic: it measures how much predictive information one attribute carries about the target label in the data distribution, and being based on mutual information, it is strictly non\-negative\. ASPG therefore cannot by itself determine whether adding a particular attribute to an LLM prompt will help or hurt simulation performance\. The performance degradation observed at some completeness levels \(e\.g\., Table[4](https://arxiv.org/html/2609.25066#S3.T4)\) is better understood as an effect of how the model jointly uses multiple attributes—individually informative attributes may still hurt performance when combined, due to redundancy, conflicting cues, longer\-context interference, or model misinterpretation—rather than as evidence that any specific attribute is intrinsically harmful\.

### 3\.4Layer III — Population Aggregation\.

In Layer III, we evaluate whether improvements in individual\-level simulation accuracy from Layer II reliably propagate to population\-level distributional alignment\. We analyze two questions: \(i\) how profile completeness gains in Layer II translate to R2\(Finding 5\), and \(ii\) how population coverage independently affects R2\(Finding 6\)\.

#### Finding 5: R1 Gains Do Not Reliably Transfer to R2

We track R2 as Layer II profile completenessccincreases in Figure[3](https://arxiv.org/html/2609.25066#S2.F3)\. R2 improves withccbut with diminishing returns: as profile completeness increases, the aggregated distribution moves closer to the empirical one, but the marginal gain shrinks asccgrows large and R1 begins to saturate\. This overall trend, however, reflects only the average trajectory and does not guarantee that each incremental step in c produces consistent gains at both levels\.

Figure 5:Population simulation reliability versus population coverage\. TVD between the aggregated simulated distribution and the target population distribution as a function of the number of simulated individualsNN\. The dashed curve shows the GT sampling baseline: human ground\-truth responses aggregated at varying sampling sizes, serving as a reference for the sampling variance of real human responses\. Theccis fixed at 100%\.Table 4:Comparison of model performance across four datasets with varying profile completenesscc\.Δ\\DeltaACC andΔ\\DeltaTVD denote the relative percentage change with respect to the previous profile\-completeness level \(i\.e\., the preceding row in each block\)\.Bluevalues mark steps where ACC increases while TVD also increases \(R1 improves, R2 worsens\);Orangevalues mark steps where both decrease \(R1 worsens, R2 improves\)\. In both cases R1 and R2 move in opposite directions, illustrating the decoupling in Finding 5\.This overall improvement trend does not mean R1 gains reliably transfer to R2\. As shown in Table[4](https://arxiv.org/html/2609.25066#S3.T4),Δ​ACC\\Delta\\mathrm\{ACC\}andΔ​TVD\\Delta\\mathrm\{TVD\}can move in opposite directions across tasks, models, and completeness levels\. We note that 26% of incremental steps in Table[4](https://arxiv.org/html/2609.25066#S3.T4)show R1 and R2 moving in opposite directions\.

To illustrate how this decoupling manifests in practice, we examine two representative cases in Table[5](https://arxiv.org/html/2609.25066#S3.T5)\. In Case 1 \(Immigration,c=12\.5%→25%c=12\.5\\%\\to 25\\%\), ACC improves but TVD worsens: as more profile information is provided, the model increasingly concentrates its predictions onQuite good\(bias:\+4\.00%→\+13\.06%\+4\.00\\%\\to\+13\.06\\%\) while further under\-predictingNeither good nor badandVery good, pushing the predicted distribution away from the true population distribution\. In Case 2 \(Media,c=66\.7%→83\.3%c=66\.7\\%\\to 83\.3\\%\), the reverse holds: the model’s over\-prediction ofBystander Entertainerand under\-prediction ofLoyal Advocateboth partially correct, bringing the predicted distribution closer to the truth, but individual accuracy drops\. Together, the cases reveal that TVD is driven by distributional shifts in a small subset of options, which can move independently of overall ACC\.

#### Finding 6: Population Coverage Reduces Variance but Not Bias

We study how population coverage affects R2 in Layer III by varying the number of simulated individualsNNand measuring TVD between the aggregated simulated distribution and the target population distribution; see Figure[5](https://arxiv.org/html/2609.25066#S3.F5)\. As a reference, we include a ground\-truth \(GT\) sampling baseline, obtained by aggregating actual human responses subsampled at varying population sizesNN; this baseline characterizes the sampling variance of real human responses and isolates the error that remains even with perfect individual\-level simulation\. This distinction between variance and bias is further corroborated in Appendix[F](https://arxiv.org/html/2609.25066#A6), where we jointly vary N and c\.

Table 5:Option\-level prediction distributions for two decoupling cases \(Qwen3\-235B\-A22B\), illustrating Finding 5\. GT% denotes the occurrence rate of each option in the ground\-truth human data\. Bias values are colored by sign:red= over\-prediction,blue= under\-prediction\.Case 1\(Immigration\):c=12\.5c\{=\}12\.5%→25\{\\to\}25%, ACC↑\\uparrow, TVD↑\\uparrow\.Case 2\(Media\):c=66\.7c\{=\}66\.7%→83\.3\{\\to\}83\.3%, ACC↓\\downarrow, TVD↓\\downarrow\.Results show that population reliability improves quickly as coverage increases and then levels off\. Model rankings are also unstable at smallNN, but become much more consistent onceNNis large enough for TVD to stabilize\. WhenNNis large, the remaining error is driven primarily by systematic distributional mismatch rather than sampling noise\. AsNNincreases, the GT sampling baseline steadily decreases and approaches zero, whereas LLM\-based simulations converge to a non\-zero plateau\. The persistent gap between the LLM curves and the GT baseline indicates that increasingNNreduces sampling variance but cannot eliminate the residual distributional mismatch\.

## 4Related Work

FindingsSimulation LayersConfigurationsEvaluationWorkF1F2F3F4F5F6IIIIIICap\.Com\.Cov\.R1R2[Hu et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib18)–R––––×\\times✓✓✓×\\times×\\times×\\times✓[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35)–R––––×\\times✓×\\times✓×\\times×\\times✓×\\times[Dominguez\-Olmedo et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib12)–R––––✓×\\times✓✓×\\times×\\times×\\times✓[Meister et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib24)––––––✓✓✓✓×\\times×\\times×\\times✓[Geng et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib15)RR––––×\\times✓✓✓✓×\\times×\\times✓[Santurkar et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib30)––––––✓✓✓✓×\\times×\\times×\\times✓[Hwang et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib21)R–––––✓✓×\\times✓✓×\\times✓×\\times[Qu and Wang \(2024\)](https://arxiv.org/html/2609.25066#bib.bib29)R–––R–×\\times✓✓✓✓×\\times✓✓ReliMap \(ours\)✓✓✓✓✓✓✓✓✓✓✓✓✓✓Table 6:Comparison with existing evaluation studies\. \(a\) Coverage of our six findings F1–F6 \(§[3](https://arxiv.org/html/2609.25066#S3)\): “R” = related prior evidence, “–” = not discussed\. \(b\) Coverage of ReliMap’s design space across simulation layers, configuration dimensions \(Cap\.: model capacity, Com\.: profile completeness, Cov\.: population coverage\), and evaluation levels \(R1/R2\)\.### 4\.1LLM\-Based Simulation Evaluation

LLM\-based human behavior simulation is increasingly used to generate survey responses, motivating benchmarks that target simulation reliability itself\.[Santurkar et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib30)and[Durmus et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib13)use real polls to test whether models match human viewpoint distributions across groups and countries\.[Hu et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib18)further treats simulation as a standalone capacity, covering diverse tasks with a unified, comparable protocol\.[Meister et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib24)evaluates 43 LLMs against the American Community Survey and reports label bias and near\-random responding, suggesting that the alignment scores may reflect closeness to uniform\. However, existing evaluations still emphasize end\-to\-end scores, offering limited insight into where biases arise and how they propagate\.

### 4\.2Individual\-level Simulation

In individual\-level simulation evaluation, prior work has explored methods for improving per\-respondent simulation accuracy\.[Bisbee et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib5)and[Tzachristas et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib32)show that zero\-shot prompting is easy to deploy but can be highly sensitive to prompt design and sampling randomness, leading to unstable per\-respondent accuracy\.[Wang et al\. \(2025a\)](https://arxiv.org/html/2609.25066#bib.bib33)further finds that prompting\-based simulation often underestimates the variance of human opinions, producing overly uniform outputs that obscure individual heterogeneity\. Beyond prompting,[Wang et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib36)studies conjoint analysis and proposes a statistical data\-augmentation framework that integrates LLM\-generated data with a small amount of human data to debias synthetic responses\. In parallel,[Suh et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib31)and[Cao et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib6)directly fine\-tune LLMs on survey data to align token\-level probabilities with the empirical distribution\. However, these efforts largely conflate reliability with end\-to\-end improvement and offer limited insight into why simulations fail; ReliMap instead decomposes simulation into layers to localize error sources and guide targeted fixes\.

### 4\.3Population\-level Distribution Alignment

Beyond individual\-level accuracy, recent work on pluralistic and distributional alignment treats group\-level preference distributions as explicit training targets to better reflect heterogeneous values across users and groups\.[Chakraborty et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib7)targets group\-level preference distributions for pluralistic alignment\.[Melnyk et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib25)enforces distributional constraints,[Poddar et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib28)models user/subgroup modes, and[Yao et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib38)adds group objectives to prevent minority collapse\. ReliMap instead diagnoses*survey\-response*distribution shifts induced and propagated by configuration choices, not distribution matching as training\.

### 4\.4Comparison with Existing Evaluations

Table[6](https://arxiv.org/html/2609.25066#S4.T6)compares ReliMap with eight closely related evaluation studies\. Some observations have prior support: profile/persona conditioning substantially affects simulation quality[Geng et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib15);[Hwang et al\. \(2023\)](https://arxiv.org/html/2609.25066#bib.bib21);[Qu and Wang \(2024\)](https://arxiv.org/html/2609.25066#bib.bib29), and profile scaling varies across models[Hu et al\. \(2025\)](https://arxiv.org/html/2609.25066#bib.bib18);[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35);[Dominguez\-Olmedo et al\. \(2024\)](https://arxiv.org/html/2609.25066#bib.bib12)\. In contrast, F3 \(concentration of predictive information\), F4 \(informativeness over attribute count\), and F6 \(variance–bias decomposition\) are not systematically studied before; F5 is partially anticipated by[Qu and Wang \(2024\)](https://arxiv.org/html/2609.25066#bib.bib29), which we turn into a layer\-wise analysis\. No existing study covers all three layers, three configuration dimensions, and both evaluation levels jointly; ReliMap unifies them to localize where reliability failures arise and propagate\.

## 5Conclusion

We introduce ReliMap, modeling LLM\-based human behavior simulation as a three\-layer generative process and evaluating reliability at individual and population levels across model capacity, profile completeness, and population coverage\. Experiments on four tasks with eleven LLMs map reliability variation across layers and configurations\. Individual\-level gains do not transfer reliably to the population level, and larger coverage reduces variance but not systematic bias; reliable simulation requires coordinated improvement across layers\.

## Limitations

- •Our study focuses on human behavior simulation in survey\-style tasks; our conclusions may not directly generalize to more complex settings such as interactive dialogue or multi\-agent dynamics\.
- •We evaluate individual\-level reliability using accuracy, which directly reflects the discrete nature of survey responses but is a winner\-takes\-all metric that does not capture the subjectivity inherent in human decision\-making\. More principled alternatives remain an open direction for future work\.
- •Our analysis is observational\. While we identify how configuration choices affect reliability, the underlying mechanisms—such as why larger models benefit more from richer profiles—remain unexplored\. Interpretability of LLM\-based simulation is an important direction for future work\.
- •Pretraining contamination cannot be fully ruled out, as the survey questions and public opinion data we use may appear in model pretraining corpora\. This concern is most relevant to Layer I, where models generally perform poorly without profile conditioning, suggesting that contamination does not drive our main conclusions, which are comparative rather than absolute\.

## References

- Aher et al\. \(2023\)Gati Aher, Rosa I\. Arriaga, and Adam Tauman Kalai\. 2023\.[Using large language models to simulate multiple humans and replicate human subject studies](https://arxiv.org/abs/2208.10264)\.*Preprint*, arXiv:2208\.10264\.
- Argyle et al\. \(2023\)Lisa P\. Argyle, Ethan C\. Busby, Nancy Fulda, Joshua R\. Gubler, Christopher Rytting, and David Wingate\. 2023\.[Out of one, many: Using language models to simulate human samples](https://doi.org/10.1017/pan.2023.2)\.*Political Analysis*, 31\(3\):337–351\.
- Association \(2022\)The World Values Survey Association\. 2022\.[Wvs database](https://www.worldvaluessurvey.org/wvs.jsp)\.
- Binz et al\. \(2025\)Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda\-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Éltető, and 1 others\. 2025\.A foundation model to predict and capture human cognition\.*Nature*, pages 1–8\.
- Bisbee et al\. \(2024\)James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson\. 2024\.Synthetic replacements for human survey data? the perils of large language models\.*Political Analysis*, 32\(4\):401–416\.
- Cao et al\. \(2025\)Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich\. 2025\.[Specializing large language models to simulate survey response distributions for global populations](https://arxiv.org/abs/2502.07068)\.*Preprint*, arXiv:2502\.07068\.
- Chakraborty et al\. \(2024\)Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang\. 2024\.[Maxmin\-rlhf: Alignment with diverse human preferences](https://arxiv.org/abs/2402.08925)\.*Preprint*, arXiv:2402\.08925\.
- Cho et al\. \(2024\)Suhyun Cho, Jaeyun Kim, and Jang Hyun Kim\. 2024\.Llm\-based doppelgänger models: leveraging synthetic data for human\-like responses in survey simulations\.*IEEE Access*\.
- Chu et al\. \(2023\)Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy\. 2023\.[Language models trained on media diets can predict public opinion](https://arxiv.org/abs/2303.16779)\.*Preprint*, arXiv:2303\.16779\.
- Comanici et al\. \(2025\)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others\. 2025\.Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.*arXiv preprint arXiv:2507\.06261*\.
- DeepSeek \(2025\)DeepSeek\. 2025\.[Deepseek\-v3\.2\-exp release \| deepseek api docs](https://api-docs.deepseek.com/news/news250929)\.
- Dominguez\-Olmedo et al\. \(2024\)Ricardo Dominguez\-Olmedo, Moritz Hardt, and Celestine Mendler\-Dünner\. 2024\.[Questioning the survey responses of large language models](https://arxiv.org/abs/2306.07951)\.*Preprint*, arXiv:2306\.07951\.
- Durmus et al\. \(2024\)Esin Durmus, Karina Nguyen, Thomas I\. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield\-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli\. 2024\.[Towards measuring the representation of subjective global opinions in language models](https://arxiv.org/abs/2306.16388)\.*Preprint*, arXiv:2306\.16388\.
- ESS ERIC \(2025\)Sikt ESS ERIC\. 2025\.[Ess11 \- integrated file, edition 4\.0 \| ess \- sikt](https://ess.sikt.no/en/datafile/242aaa39-3bbb-40f5-98bf-bfb1ce53d8ef)\.
- Geng et al\. \(2024\)Mingmeng Geng, Shiti He, and Roberto Trotta\. 2024\.Are large language models chameleons?In*ICML 2024 Workshop on LLMs and Cognition*\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others\. 2025\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*\.
- Hewitt et al\. \(2024\)Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer\. 2024\.Predicting results of social science experiments using large language models\.*Preprint*\.
- Hu et al\. \(2025\)Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger\. 2025\.[Simbench: Benchmarking the ability of large language models to simulate human behaviors](https://arxiv.org/abs/2510.17516)\.*Preprint*, arXiv:2510\.17516\.
- Hu et al\. \(2024\)Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek\. 2024\.[Generative language models exhibit social identity biases](https://arxiv.org/abs/2310.15819)\.*Preprint*, arXiv:2310\.15819\.
- Huang et al\. \(2025\)Ji Huang, Mengfei Li, and Shuai Shao\. 2025\.[Distribution shift alignment helps llms simulate survey response distributions](https://arxiv.org/abs/2510.21977)\.*Preprint*, arXiv:2510\.21977\.
- Hwang et al\. \(2023\)EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon\. 2023\.Aligning language models to user opinions\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 5906–5919\.
- Kolluri et al\. \(2025\)Akaash Kolluri, Shengguang Wu, Joon Sung Park, and Michael S\. Bernstein\. 2025\.[Finetuning llms for human behavior prediction in social science experiments](https://arxiv.org/abs/2509.05830)\.*Preprint*, arXiv:2509\.05830\.
- Manning et al\. \(2024\)Benjamin S\. Manning, Kehang Zhu, and John J\. Horton\. 2024\.[Automated social science: Language models as scientist and subjects](https://arxiv.org/abs/2404.11794)\.*Preprint*, arXiv:2404\.11794\.
- Meister et al\. \(2024\)Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto\. 2024\.[Benchmarking distributional alignment of large language models](https://arxiv.org/abs/2411.05403)\.*Preprint*, arXiv:2411\.05403\.
- Melnyk et al\. \(2024\)Igor Melnyk, Youssef Mroueh, Brian Belgodere, Mattia Rigotti, Apoorva Nitsure, Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jerret Ross\. 2024\.[Distributional preference alignment of llms via optimal transport](https://arxiv.org/abs/2406.05882)\.*Preprint*, arXiv:2406\.05882\.
- Moon et al\. \(2024\)Suhong Moon, Marwa Abdulhai, Minwoo Kang, Joseph Suh, Widyadewi Soedarmadji, Eran Kohen Behar, and David M\. Chan\. 2024\.[Virtual personas for language models via an anthology of backstories](https://arxiv.org/abs/2407.06576)\.*Preprint*, arXiv:2407\.06576\.
- OpenAI \(2025\)OpenAI\. 2025\.[Introducing gpt\-4\.1 in the api \| openai](https://openai.com/index/gpt-4-1/)\.
- Poddar et al\. \(2024\)Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques\. 2024\.[Personalizing reinforcement learning from human feedback with variational preference learning](https://arxiv.org/abs/2408.10075)\.*Preprint*, arXiv:2408\.10075\.
- Qu and Wang \(2024\)Yanchen Qu and Jian Wang\. 2024\.Performance and biases of large language models in public opinion simulation\.*Humanities and Social Sciences Communications*, 11\(1\):1–13\.
- Santurkar et al\. \(2023\)Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto\. 2023\.[Whose opinions do language models reflect?](https://arxiv.org/abs/2303.17548)*Preprint*, arXiv:2303\.17548\.
- Suh et al\. \(2025\)Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang\. 2025\.[Language model fine\-tuning on scaled survey data for predicting distributions of public opinions](https://arxiv.org/abs/2502.16761)\.*Preprint*, arXiv:2502\.16761\.
- Tzachristas et al\. \(2025\)Ioannis Tzachristas, Santhanakrishnan Narayanan, and Constantinos Antoniou\. 2025\.Guided persona\-based ai surveys: Can we replicate personal mobility preferences at scale using llms?*arXiv preprint arXiv:2501\.13955*\.
- Wang et al\. \(2025a\)Angelina Wang, Jamie Morgenstern, and John P Dickerson\. 2025a\.Large language models that replace human participants can harmfully misportray and flatten identity groups\.*Nature Machine Intelligence*, pages 1–12\.
- Wang et al\. \(2025b\)Han Wang, Jacek Pawlak, and Aruna Sivakumar\. 2025b\.Can large language models simulate human responses? a case study of stated preference experiments in the context of heating\-related choices\.*arXiv preprint arXiv:2503\.10652*\.
- Wang et al\. \(2025c\)Jia Wang, Ziyu Zhao, Tingjuntao Ni, and Zhongyu Wei\. 2025c\.[Sociobench: Modeling human behavior in sociological surveys with large language models](https://arxiv.org/abs/2510.11131)\.*Preprint*, arXiv:2510\.11131\.
- Wang et al\. \(2024\)Mengxin Wang, Dennis J Zhang, and Heng Zhang\. 2024\.Large language models for market research: A data\-augmentation approach\.*arXiv preprint arXiv:2412\.19363*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yao et al\. \(2025\)Binwei Yao, Zefan Cai, Yun\-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, and Junjie Hu\. 2025\.[No preference left behind: Group distributional preference optimization](https://arxiv.org/abs/2412.20299)\.*Preprint*, arXiv:2412\.20299\.

## Appendix APotential Risks

This work examines LLM\-based human behavior simulation in survey\-style tasks and introduces ReliMap to diagnose reliability at both the individual and population levels\. We use established survey datasets, collect no new personal data, and report results only in aggregate, which helps reduce privacy risks\. Still, simulated responses can inherit biases from the underlying data and models, so they should not be interpreted as real public opinion\. ReliMap is meant to support evaluation and auditing, rather than high\-stakes use\.

## Appendix BTheoretical Insight

We provide theoretical insights into when and how the reliability of LLM\-based human behavior simulation can converge to its ideal limit\. Rather than viewing reliability as a single score, we show that it emerges from a multi\-layer structure\.

#### Multi\-layer structure

Letssdenote a task scenario,uuan individual,xxan individual profile, andyya behavioral outcome\. Human behavior is assumed to arise from a structured process\. The target behavior distribution under scenariossis

p∗​\(y∣s\)=∑up∗​\(u\)​∑xp∗​\(x∣u\)​p∗​\(y∣s,x,u\)\.p^\{\\ast\}\(y\\mid s\)=\\sum\_\{u\}p^\{\\ast\}\(u\)\\sum\_\{x\}p^\{\\ast\}\(x\\mid u\)\\,p^\{\\ast\}\(y\\mid s,x,u\)\.\(5\)An LLM\-based simulator induces an approximate distribution of the same form,

p⁡\(y∣s\)=∑up⁡\(u\)​∑xp⁡\(x∣u\)​p​\(y∣s,x,u\),p\(y\\mid s\)=\\sum\_\{u\}p\(u\)\\sum\_\{x\}p\(x\\mid u\)\\,p\(y\\mid s,x,u\),\(6\)where deviations arise from imperfect modeling of individual sampling, profile construction, and scenario\-conditioned behavior generation\.

#### Reliability gap\.

We define the population\-level discrepancy under scenariossas theℓ1\\ell\_\{1\}distance

D\(s\)≜∥p∗\(⋅∣s\)−p\(⋅∣s\)∥1\.D\(s\)\\triangleq\\bigl\\\|p^\{\\ast\}\(\\cdot\\mid s\)\-p\(\\cdot\\mid s\)\\bigr\\\|\_\{1\}\.\(7\)

#### Layer\-wise error propagation\.

Consider a single\(u,x\)\(u,x\)term and define

A∗\\displaystyle A^\{\\ast\}=p∗​\(y∣s,x,u\),\\displaystyle=p^\{\\ast\}\(y\\mid s,x,u\),\(8\)B∗\\displaystyle B^\{\\ast\}=p∗​\(x∣u\),\\displaystyle=p^\{\\ast\}\(x\\mid u\),C∗\\displaystyle C^\{\\ast\}=p∗​\(u\)\.\\displaystyle=p^\{\\ast\}\(u\)\.and their simulator counterparts

A=p⁡\(y∣s,x,u\),B=p⁡\(x∣u\),C=p⁡\(u\)\.A=p\(y\\mid s,x,u\),\\quad B=p\(x\\mid u\),\\quad C=p\(u\)\.\(9\)The discrepancy contributed by this term is\|A∗​B∗​C∗−A​B​C\|\\bigl\|A^\{\\ast\}B^\{\\ast\}C^\{\\ast\}\-ABC\\bigr\|\. By adding and subtracting intermediate terms and applying the triangle inequality, we have

\|A∗​B∗​C∗−A​B​C\|≤\\displaystyle\|A^\{\\ast\}B^\{\\ast\}C^\{\\ast\}\-ABC\|\\leq\|B∗​C∗\|​\|A∗−A\|\\displaystyle\|B^\{\\ast\}C^\{\\ast\}\|\\,\|A^\{\\ast\}\-A\|\(10\)\+\|A​C∗\|​\|B∗−B\|\\displaystyle\+\|AC^\{\\ast\}\|\\,\|B^\{\\ast\}\-B\|\+\|A​B\|​\|C∗−C\|\.\\displaystyle\+\|AB\|\\,\|C^\{\\ast\}\-C\|\.

#### Bounding the overall error\.

For any fixedyy, the difference between the target and simulated probabilities can be written as

p∗​\(y∣s\)−p⁡\(y∣s\)=∑u∑x\(A∗​B∗​C∗−A​B​C\)\.p^\{\\ast\}\(y\\mid s\)\-p\(y\\mid s\)=\\sum\_\{u\}\\sum\_\{x\}\\bigl\(A^\{\\ast\}B^\{\\ast\}C^\{\\ast\}\-ABC\\bigr\)\.\(11\)Applying the triangle inequality and summing overyyyields

D⁡\(s\)\\displaystyle D\(s\)=∑y\|p∗​\(y∣s\)−p⁡\(y∣s\)\|\\displaystyle=\\sum\_\{y\}\\bigl\|p^\{\\ast\}\(y\\mid s\)\-p\(y\\mid s\)\\bigr\|\(12\)≤∑y,u,x\|A∗​B∗​C∗−A​B​C\|\.\\displaystyle\\leq\\sum\_\{y,u,x\}\\bigl\|A^\{\\ast\}B^\{\\ast\}C^\{\\ast\}\-ABC\\bigr\|\.
Since all terms are probabilities in\[0,1\]\[0,1\], we can further upper bound each term by

\|A∗​B∗​C∗−A​B​C\|≤\|A∗−A\|\+\|B∗−B\|\+\|C∗−C\|\.\|A^\{\\ast\}B^\{\\ast\}C^\{\\ast\}\-ABC\|\\leq\|A^\{\\ast\}\-A\|\+\|B^\{\\ast\}\-B\|\+\|C^\{\\ast\}\-C\|\.\(13\)
which gives the overall bound

D⁡\(s\)≤∑y∑u∑x\(\|A∗−A\|\+\|B∗−B\|\+\|C∗−C\|\)\.D\(s\)\\leq\\sum\_\{y\}\\sum\_\{u\}\\sum\_\{x\}\\Bigl\(\\lvert A^\{\\ast\}\-A\\rvert\+\\lvert B^\{\\ast\}\-B\\rvert\+\\lvert C^\{\\ast\}\-C\\rvert\\Bigr\)\.\(14\)

#### Convergence conditions\.

The bound implies thatD⁡\(s\)→0D\(s\)\\to 0is guaranteed when the following component distributions converge:

p⁡\(u\)\\displaystyle p\(u\)→p∗​\(u\),\\displaystyle\\to p^\{\\ast\}\(u\),\(15\)p⁡\(x∣u\)\\displaystyle p\(x\\mid u\)→p∗​\(x∣u\),\\displaystyle\\to p^\{\\ast\}\(x\\mid u\),p⁡\(y∣s,x,u\)\\displaystyle p\(y\\mid s,x,u\)→p∗​\(y∣s,x,u\)\.\\displaystyle\\to p^\{\\ast\}\(y\\mid s,x,u\)\.
This result shows that reliable simulation cannot be attained by optimizing any single component in isolation\. Instead, reliability emerges from coordinated convergence across individual sampling, profile construction, and scenario\-conditioned behavior generation\.

## Appendix CData Construction & Statistics

We conduct experiments on four tasks across two settings\. The first setting comprises three established survey tasks:partisan preference,immigration attitude, andreligious stance\. Task definitions are drawn from the European Social Survey \(ESS;Party\)[ESS ERIC \(2025\)](https://arxiv.org/html/2609.25066#bib.bib14), the World Values Survey \(WVS;Immigration\)[Association \(2022\)](https://arxiv.org/html/2609.25066#bib.bib3), and SocioBench \(Religion\)[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35)\. For each dataset, we follow a three\-step process\. First, we select the survey questions that define the simulation tasks\. Second, we preprocess the raw survey data by merging demographic attributes with historical survey responses to construct individual profiles\. Finally, we filter the population to retain only respondents with complete and valid profile fields\. Table[7](https://arxiv.org/html/2609.25066#A3.T7)reports statistics of the remaining valid data and the associated survey question for each task\. Table[8](https://arxiv.org/html/2609.25066#A3.T8)lists the response options for each survey question\.

The second setting introduces a real\-world dynamic event: public attitude toward the Luckin Coffee stock collapse\. It is in Chinese, further testing whether our findings generalize across languages and beyond traditional survey scenarios\. Raw data were sourced from a publicly available social media dataset\.666[https://giser2000\.github\.io/geodata\.github\.io/\#download](https://giser2000.github.io/geodata.github.io/#download)\.We filtered it for original posts containing the hashtag “\#Luckin Coffee Collapse” during the event period\. Ground\-truth labels were obtained by applying GPT\-4o as an annotator to map users’ original posts to the predefined attitude categories\. Note that GPT\-4o is used only for annotation, not for generating the target behavior: the target evidence consists of real user\-authored posts about the event, so this construction is not circular\. To verify label quality, we manually inspected 20 annotated examples and found 100% annotation accuracy on this subset\. Individual profiles were constructed from each user’s 30 most recent historical posts prior to the event\. Table[7](https://arxiv.org/html/2609.25066#A3.T7)includes statistics for this task alongside the three surveys\.

Table 7:Task settings and question texts\.Party \(options\)Immigration \(options\)Religious \(options\)Media \(options\)1\. CDU/CSU
2\. SPD
3\. The Left \(Die Linke\)
4\. Alliance 90/The Greens
5\. FDP
6\. AFD
7\. Free Voters
8\. dieBasis
9\. Die PARTEI1\. Very bad
2\. Quite bad
3\. Neither good nor bad
4\. Quite good
5\. Very good1\. Far too much power
2\. Too much power
3\. About the right amount
4\. Too little power
5\. Far too little power1\. Loyal Advocate
2\. Opportunist
3\. Moral Outrage
4\. Rational Evaluator
5\. Bystander EntertainerTable 8:Answer options for each task\.Instruction:You are participating in the International Social Survey Programme\. Assume the role of a real individual with the following personal information\. Fully immerse yourself in this persona and answer the question truthfully, based solely on the provided personal information and your previous answers to prior questions\.Personal Information and Previous Answers:\{attributes\}Question:\{question\}Options:\{options\}JSON Output:[⬇](data:text/plain;base64,eyJyZWFzb24iOiAiIiwgIm9wdGlvbiI6ICIifQ==)\{"reason":"","option":""\}Requirements:1\.Answer based on personal information only; 6–10 sentence justification\.2\.Choose best option; respond with number only\.Figure 6:Prompt template for Party, Immigration, Religion\.Instruction:You are a user attitude simulation model\. Based on historical Weibo posts provided, simulate this user’s likely attitude toward “Luckin Coffee’s stock price plummeted” and determine which category best matches\.Five Attitude Categories:1\.Loyal Advocacy:High emotional support; views Luckin as “pride of domestic brands”; heartbreak/defensiveness on stock crash without blaming company\.2\.Opportunistic Exploitation:No loyalty; only cares about prices/discounts/investment; neutral on fraud; motivation: “getting a bargain” or “buying the dip\.”3\.Moral Indignation:Strong moral outrage on corporate integrity; challenges “national brand” narrative; fraud is unforgivable\.4\.Belief Evaluation:Distinguishes product from corporate behavior; may acknowledge quality but does not forgive fraud; rational, analytical stance\.5\.Entertainment Spectator:No substantive stance; memes/jokes/viral quips \(e\.g\., “Sauce\-flavored Latte becomes Limit\-down Latte”\); lighthearted, social interaction focused\.Input:\{posts\}→\\rightarrowEvent:Luckin Coffee’s stock price plummetedOutput Rules:1\.Output exactly one valid JSON object only, no extra text/spaces/line breaks/Markdown\.2\.JSON must contain:analysis\(string\) andattitude\_id\(integer 1–5\)\.3\.Do NOT use‘‘‘jsonor code block markers\. Output raw JSON directly\.JSON Output:[⬇](data:text/plain;base64,eyJhbmFseXNpcyI6ICIiLCAiYXR0aXR1ZGVfaWQiOiAwfQ==)\{"analysis":"","attitude\_id":0\}Figure 7:Prompt template for Media Task \(translated from Chinese\)\.
## Appendix DExperimental Details

### D\.1Prompt Templates For Prediction

Figure[6](https://arxiv.org/html/2609.25066#A3.F6)shows the prompt template used to elicit survey\-style responses in the International Social Survey Programme setting\. Our design follows prior prompt\-based survey simulation setups in[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.25066#bib.bib35)\. Each instance provides \(i\) an instruction to role\-play as a real individual, \(ii\) the respondent’s personal attributes and any previous answers, \(iii\) the target question and its candidate options, and \(iv\) an explicit JSON output schema\. We require the model to justify its choice in 6–10 sentences based only on the provided attributes, and to return the selected option as a number in theoptionfield\. This structured format standardizes generation across models and facilitates automatic parsing and evaluation\. The prompt design for the Media task follows a similar structure, as shown in Figure[7](https://arxiv.org/html/2609.25066#A3.F7)\.

### D\.2Generation Settings

We employ greedy decoding across all models\. For Qwen3\-8B, Qwen3\-14B, and Qwen3\-32B, we activate thethinkmode viaenable\_think=True\. In profile\-completeness experiments, we evaluate each completenessccacross5 random seeds, with each seed generating a distinct subset of profile attributes\. Similarly, population\-coverage experiments use5 random seedsfor stochastic population sampling\. All reported results represent averages over these replications\.

## Appendix EIllustrative Example of the Simulation Pipeline

To clarify how the formal model is instantiated in practice, we provide a concrete example from the Party task\. Table[9](https://arxiv.org/html/2609.25066#A5.T9)shows how the three key components—scenarioss, individualuu, and profilexx—are populated in a real simulation instance\.

In this framework,uurefers to a specific survey respondent \(e\.g\., Respondent\_1\)\. The scenariossdefines the task context, including the question and candidate options\. The profilexxis a subset of attributes sampled from the full set of attributes available foruu, which corresponds top⁡\(x∣u\)p\(x\\mid u\)in our formulation\. Studying how simulation performance varies as we include more or fewer attributes fromxxallows us to quantify the effect of profile completeness on reliability\.

Note that two users with identical attribute setsxxwill indeed generate identical prompts\. In practice, however, since we employ a rich set of high\-dimensional sociodemographic features, cases where two distinct individuals share the sameuuare extremely rare in our dataset and can be considered negligible\.

Table 9:A concrete example of the simulation pipeline for the Party task, illustrating howss,uu, andxxare instantiated in practice\.
## Appendix FMore Experiments

### F\.1ASPG Distribution

Figure 8:Distribution of ASPG values across attributes for each task\. Most attributes carry low predictive information individually, while a small minority contributes disproportionately large predictive gain\.Figure[8](https://arxiv.org/html/2609.25066#A6.F8)shows the distribution of ASPG values across attributes for each task\. In all three tasks, the distribution is strongly right\-skewed: most attributes carry low predictive information individually, while a small minority contributes disproportionately large predictive gain\.

### F\.2Effect of Population Coverage and Profile Completeness on Distributional Bias

Figure 9:TVD Scaling with Population CoverageNNunder Varying Profile Completenesscc\.To further test ReliMap’s core claim, we ask whether population\-level distribution errors mainly reflect*estimation noise*from limited population coverage or*systematic bias*from insufficient profile information\. In the Party scenario, we vary population coverageNNwhile fixing profile completenesscc, and measure distributional deviation by TVD\. Results are reported in Figure[9](https://arxiv.org/html/2609.25066#A6.F9)\.

Across models, TVD decreases rapidly asNNincreases and then quickly plateaus, indicating that largerNNprimarily improves stability by reducing variance\. In contrast, increasingccproduces a more consistent downward shift of the curves, suggesting that the remaining gap is dominated by systematic bias that cannot be removed by sampling more agents alone\. Despite differences in absolute performance and smoothness across models, the trend is robust and aligns with ReliMap:NNmainly controls estimation variance, whereasccmore directly determines distributional alignment\. These results motivate future work on constructing higher\-information profiles and explicitly optimizing distribution\-level objectives to reduce residual bias\.

## Appendix GAlternative Metrics for R2

Besides TVD, we also computed KL divergence, Jensen–Shannon divergence \(JSD\), and Earth Mover’s Distance \(EMD\) for R2 and checked whether the main R2 findings remain visible under these alternative metrics\. Table[10](https://arxiv.org/html/2609.25066#A7.T10)reports the four metrics under varying profile completenesscc, and Table[11](https://arxiv.org/html/2609.25066#A7.T11)reports them under varying population coverageNN\(both on the Party task\)\.

The trends are consistent across all four metrics\. For profile completeness, all metrics show the same pattern: profile conditioning usually reduces population\-level mismatch relative to no conditioning \(c=0c=0\), while local non\-monotonic changes persist across completeness levels, consistent with Finding 5\. For population coverage, all four metrics decrease asNNincreases, with most gains appearing at smallerNNbefore the curves stabilize, consistent with Finding 6\.

Table 10:R2 metrics under varying profile completenessccon the Party task\. All four metrics exhibit the same qualitative trends as TVD\.Table 11:R2 metrics under varying population coverageNNon the Party task\. All four metrics decrease asNNincreases and stabilize at largerNN, consistent with the TVD\-based results\.
## Appendix HRobustness Analysis

To examine whether our findings are sensitive to implementation\-level choices, we conduct robustness tests on prompt design and decoding temperature using the Party task across three models\. Each setting is evaluated over 5 independent sampling runs, with results reported as mean individual\-level simulation accuracy \(ACC\)\.

### H\.1Prompt Variations

We test two additional prompt variants alongside the original prompt described in Appendix C\.

Original promptfollows the role\-playing instruction described in Appendix C, asking the model to fully immerse itself in the respondent’s persona\.

Prompt\_v1instructs the model to predict the respondent’s most probable choice instead of role\-playing:“You are analyzing responses for the International Social Survey Programme\. Given the demographic and attitudinal profile below, predict the most likely response this individual would provide to the survey question\.”

Prompt\_v2directly instructs the model to adopt the role of the respondent with a minimal instruction:“You’re taking part in an international survey\. Answer as a real person with these traits:”

### H\.2Temperature

We test decoding temperaturet=0\.6t=0\.6alongside the default greedy decoding \(t=0t=0\) using the original prompt template\.

Table 12:Individual\-level simulation accuracy \(ACC, %\) under different prompt variants and decoding temperatures on the Party task, supporting the robustness of Findings 1 and 2\. Original refers to the default prompt used in the main experiments\.Across all prompt variants and temperature settings, two patterns remain consistent\. First, individual\-level simulation accuracy improves with profile completeness but with diminishing marginal gains, confirming the robustness of Finding 1\. Second, larger models achieve higher accuracy and continue to benefit more from richer profiles, confirming the robustness of Finding 2\. While absolute accuracy varies across settings, the macro\-level patterns identified in our main experiments remain stable, suggesting that our findings are not artifacts of particular implementation choices\.

## Appendix IStatistical Uncertainty Analysis

To assess the statistical reliability of our experimental findings, we report the mean and standard deviation of the individual\-level accuracy \(R1\) across 5 fixed random seeds \(2025, 2026, 2027, 2028, 2029\), where randomness stems from the attribute subset sampling procedure\.

Figure[10](https://arxiv.org/html/2609.25066#A9.F10)replicates Figure 3\(a\) across all four tasks \(Party, Immigration, Religion, and Media\) with±\\pm1 standard deviation shaded regions\. The shaded regions reflect the variability introduced by different random attribute subset samples rather than model stochasticity\. As shown, while some tasks \(e\.g\., Religion\) exhibit larger variance due to the smaller dataset size, the main trends remain consistent across models and tasks, supporting the reliability of our conclusions\.

Figure 10:Individual\-level accuracy \(R1\) as a function of profile completenessccacross all four tasks, with±\\pm1 standard deviation shaded regions computed over 5 fixed random seeds \(2025–2029\)\. Variance stems from attribute subset sampling\. Despite variability in some tasks, the overall trends remain consistent across models\.
## Appendix JUse of AI Assistant

In this paper, we only use ChatGPT777[https://chatgpt\.com/](https://chatgpt.com/)for grammar proofreading and spell checking\.