# Population Fidelity：评估 LLM 中的人口代表性

# Population Fidelity: Evaluating Population Representativeness in LLMs
Source: [https://arxiv.org/html/2609.36253](https://arxiv.org/html/2609.36253)
Neemias B\. da SilvaAffiliation:University of Toronto, CanadaAffiliation:Federal University of Technology – Paraná, BrazilEmail:[neemias\.buceli@mail\.utoronto\.ca](mailto:[email protected])Ali SutaniAffiliation:University of Toronto, CanadaEmail:[abhishek\.moturu@mail\.utoronto\.ca](mailto:[email protected])Abhishek MoturuAffiliation:University of Toronto, CanadaEmail:[harris\.yang@mail\.utoronto\.ca](mailto:[email protected])Harris YangAffiliation:University of Toronto, CanadaEmail:[martin\.lukk@utoronto\.ca](mailto:[email protected])Daniel SilverAffiliation:University of Toronto, CanadaEmail:[dan\.silver@utoronto\.ca](mailto:[email protected])Matt RattoAffiliation:University of Toronto, CanadaEmail:[matt\.ratto@utoronto\.ca](mailto:[email protected])Thiago H\. SilvaAffiliation:University of Toronto, CanadaAffiliation:Federal University of Technology – Paraná, BrazilEmail:[th\.silva@utoronto\.ca](mailto:[email protected])

###### Abstract

Large language models \(LLMs\) show considerable potential in simulating human attitudes and preferences\. Prior work finds that LLM\-generated responses can compress the range of attitudes found within populations and misrepresent particular subgroups in ways that vary across models and topics\. We introduce Population Fidelity, an evaluation framework that distinguishes key conditions required for a set of LLM\-generated responses to represent a population\. It incorporates three dimensions: group\-level accuracy, the amount of between\-group variation, and the structure of that variation\. We demonstrate the framework’s utility in two ways\. First, we reproduce a prior study of “machine bias” in LLM survey responses and apply the framework to its models and more recent ones, showing that poor representation reflects not only insufficient between\-group variation but also variation assigned to the wrong groups\. Second, we evaluate one proposed approach to improving models’ population representativeness: cultural fine\-tuning\. We find that cultural fine\-tuning can improve alignment with the survey center without improving the representation of within\-population differences, a distinction that measures of aggregate agreement do not capture\. We argue that representing a population requires models to reproduce several features of human attitudinal variation simultaneously\. Our framework organizes these features and provides reusable code, data, and trained models for evaluating population fidelity across substantive domains and assessing proposed alignment methods\.

## 1Introduction

Large language models \(LLMs\) show considerable potential in simulating human attitudes, behaviors, and preferences in wide\-ranging applied and research contexts\([Bail, 2024](https://arxiv.org/html/2609.36253#bib.bib4);[Davidson & Karell, 2025](https://arxiv.org/html/2609.36253#bib.bib9);[Karetnikov et al\., 2026](https://arxiv.org/html/2609.36253#bib.bib16)\)\. There is particular interest in using LLMs to generate synthetic survey responses as proxies for human responses\. With appropriate prompting and context, the resulting “silicon samples” should reproduce the response distributions of human populations\([Argyle et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib2);[Dillion et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib12)\)\. This could enable inferences about population\-level attitudes on important social, cultural, and political issues, especially as traditional survey methods continue to face threats to their reliability\([Keeter et al\., 2017](https://arxiv.org/html/2609.36253#bib.bib17);[Zhang et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib32)\)\.

In practice, researchers have raised serious concerns about LLMs’ ability to reproduce human response patterns\. Several studies, focusing on GPT models, have documented how LLMs reproduce only a limited range of attitudinal variation and misrepresent the positions of particular groups and populations as a whole\([Bisbee et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib5);[Santurkar et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib25)\)\. Recently,[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)documented a broader pattern of “machine bias,” whereby LLMs’ responses show unrealistically limited variation across demographic groups and misrepresent human responses in inconsistent ways\. Rather than LLMs as a whole being systematically biased toward particular demographic groups, this finding highlights how patterns of \(mis\-\)representation vary across models and topics\. This raises the question of how to systematically evaluate such differences across LLMs\. This is particularly urgent given that researchers and application developers must choose from among a wide range of models, with distinct representational capabilities, when implementing different types of social simulations\.

Model selection is further complicated by proposed methods for improving LLMs’ population representativeness\. One approach involves fine\-tuning language models on empirical data to better reproduce observed response distributions, for tasks including missing data imputation in surveys and predicting responses to unasked questions\([Kim & Lee, 2026](https://arxiv.org/html/2609.36253#bib.bib18);[Suh et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib26)\)\. CultureLLM, for example, uses cross\-national World Values Survey \(WVS\) responses for fine\-tuning culture\-specific LLMs that address foundation models’ knowledge deficiencies about cultures and languages underrepresented in training corpora\([Li et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib20)\)\. These approaches often target specific forms of alignment, such as matching a model’s responses to a*country\-level*response distribution for a set of survey questions\([Cao et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib7)\)\. While such fine\-tuning can bring model responses closer to those of a target population, it does not necessarily reproduce variation*within*that population and among its demographic subgroups\([Wightman et al\., 2026](https://arxiv.org/html/2609.36253#bib.bib30)\)\. It may even compress or misplace differences among social groups, exacerbating existing sources of misalignment\([Adilazuarda et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib1)\)\. Recent work cautions against inferring alignment from marginal response distributions, which may obscure how models misrepresent populations’ underlying attitudinal structures\([Williams et al\., 2026](https://arxiv.org/html/2609.36253#bib.bib31)\)\. We contribute to this line of research with a framework for assessing how well models recover the magnitude and structure of between\-group differences within survey items, properties that measures of item\-level agreement and cross\-item correlation do not directly assess\.

We introduce Population Fidelity, an evaluation framework that distinguishes key conditions required for a set of LLM\-generated responses to represent a population\. It decomposes population representativeness into three dimensions\.*Accuracy*measures model responses’ correspondence with each population subgroup’s observed response distribution\.*Adaptability*measures whether the model reproduces the observed amount of variation between subgroups\.*Structure*measures whether the same groups are relatively similar or different in generated and observed responses\. The Population Fidelity Score \(PFS\) combines accuracy, adaptability, and structure, providing a summary measure of model representativeness in relation to a given set of survey responses\. Considering the overall score alongside its components and*center alignment*\(a separate measure of agreement between model and survey centers\) demonstrates how LLMs can achieve some aspects of representativeness despite failing to achieve overall fidelity and how better center alignment need not imply better representation of within\-population variation\.

Figure 1:Stylized example with three subpopulations \(A–C\)\. Colored curves show each group’s share of respondents and sum to the pooled distribution \(gray\), which is identical in every panel \(Scenter=1S\_\{\\mathrm\{center\}\}=1for all models\); dashed lines mark group means\. Calculations are in Appendix[B\.3](https://arxiv.org/html/2609.36253#A2.SS3)\.Figure[1](https://arxiv.org/html/2609.36253#S1.F1)illustrates why aggregate agreement alone is insufficient to establish Population Fidelity\. Each model\-generated distribution appears representative: pooled across groups, each reproduces the survey’s overall response distribution exactly \(the gray area in every panel\), and the three have nearly identical accuracy\. Yet each misrepresents between\-group differences: M1 eliminates them by giving every group the same distribution; M2 exaggerates them, making groups appear farther apart than observed; and M3 reproduces the correct amount of difference but assigns it to the wrong groups\. Center alignment and accuracy therefore cannot distinguish these models, whereas adaptability and structure can\. M1 and M3 receivePFS=0\\mathrm\{PFS\}=0for different reasons, while M2, which preserves which groups are more alike, is penalized mainly for excess variation\.

We demonstrate the framework’s utility in two ways\. First, we reproduce the results of[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)from their original replication materials and apply the framework to both their models and newer ones\. Their diagnosis centers on too little variation across demographic groups; incorporating structure shows that this is only part of the problem\. Across 164 evaluated combinations of model variant, survey question, and elicitation method, median accuracy is approximately0\.830\.83, whereas median structure is approximately0\.060\.06and is the limiting PFS component in 159 cases, and even models that match or exceed the survey’s between\-group variation rarely reproduce its structure\. Second, we evaluate one proposed approach to improving models’ population representativeness: cultural fine\-tuning\. Specifically, we apply CultureLLM\-style fine\-tuning\([Li et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib20)\)and compare German and Mexican adapted models with their corresponding as\-released models\. CultureLLM\-style fine\-tuning does not improve PFS on average: mean changes are−0\.024\-0\.024for German and−0\.014\-0\.014for Mexican fine\-tuning\. Within Mexico, Mexican fine\-tuning improves center alignment by0\.0380\.038while target\-country PFS changes by−0\.010\-0\.010\. These results show that fine\-tuning can improve alignment with a target survey center without improving the representation of differences within that population, a distinction that measures of aggregate agreement overlook\.

Overall, we argue that representing a population requires models to reproduce several features of human attitudinal variation simultaneously\. Our framework organizes these dimensions and we provide reusable code and data for evaluating population fidelity and alignment methods at[https://github\.com/CriticalMaking/LLM\-population\-fidelity](https://github.com/CriticalMaking/LLM-population-fidelity)\.

## 2Related Work

### 2\.1Misalignment in LLM\-simulated human attitudes

Despite its considerable potential\([Argyle et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib2)\), recent work has highlighted serious limitations in LLMs’ abilities to generate synthetic survey responses that faithfully represent human populations\. Although models can reproduce human attitudes at the coarsest levels of analysis, including average responses that are close to survey\-based population averages, they fail to reproduce much of the underlying variation within populations, including the true range of positions expressed and their relationships to demographic characteristics\([Bisbee et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib5)\)\. Recent studies document several kinds of misalignment in LLM\-generated responses, including bias towards dominant viewpoints and flattening the perspectives of smaller countries and population subgroups\([Santurkar et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib25);[Wang et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib29)\)\.[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)identify a broader pattern of “machine bias” across a range of models, referring to their tendency to not only misrepresent population groups but to do so inconsistently, favoring different demographic groups in ways that vary arbitrarily across models and topics\. These biases constitute potential sources of harm for misrepresented groups\([Gallegos et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib13)\), with concerning downstream consequences given the wide\-ranging and increasingly consequential scenarios in which LLMs are deployed\([Cheng et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib8);[Lukk, 2026](https://arxiv.org/html/2609.36253#bib.bib22)\)\. Because models’ representational capabilities vary across models and questions, it is crucial to develop evaluations that assess LLMs’ performance on several key aspects of population representativeness\.

### 2\.2Cultural Alignment and Steering Methods

Researchers have proposed several ways to steer models to improve alignment with cultural and population targets\. A variety of studies investigate the effects of techniques applied at inference time, most notably demographic persona\-based prompting\([Argyle et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib2);[Liu et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib21)\), with evidence that these do not significantly improve model representativeness\([Santurkar et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib25);[Bisbee et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib5)\)and may even worsen certain kinds of social biases\([Deshpande et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib10);[Wan et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib28)\)\. An alternative approach involves fine\-tuning language models on relevant empirical data to better reproduce real\-world responses or predict responses on unasked survey questions\([Kim & Lee, 2026](https://arxiv.org/html/2609.36253#bib.bib18)\)\. CultureLLM uses selected World Values Survey questions as seed data to generate semantically equivalent training examples and fine\-tune models aligned with particular cultural perspectives, especially those likely to be underrepresented in model training data\([Li et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib20)\)\.[Cao et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib7)instead fine\-tune models to reproduce country\-level response distributions, while[Suh et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib26)train on distributions for demographic and ideological subpopulations\. These promising approaches encode different notions of what successful population adaptation entails, further motivating a framework that distinguishes the aspects of population representation recovered by different alignment methods\.

### 2\.3Evaluating Population Representation

Previous works primarily use single\-metric evaluations to compare model responses to those of human survey respondents, including correlation and regression coefficients for individual quantitative outcomes\([Argyle et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib2);[Bisbee et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib5)\)and distance measures of pairs of opinion distributions\([Santurkar et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib25);[Boelaert et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib6)\)\. An exception to this is work by[Williams et al\. \(2026\)](https://arxiv.org/html/2609.36253#bib.bib31), who, in addition to comparing simulated and observed response distributions, measure whether models accurately reproduce how human attitudes are structured*across*survey questions\. In contrast to these approaches, we enumerate multiple conditions required for a set of LLM\-generated responses to represent a population, with an emphasis on faithful variation among social groups*within*a given question\. We develop metrics to assess those conditions and propose a summary Population Fidelity Score to enable comparisons across models and survey items\.

## 3Machine Bias Design and Reproduction

Our evaluation uses the survey data, demographic cells, prompts, and distance measure of[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)\. Their design compares LLM responses with three waves of the WVS across happiness, political ideology, religious attendance, and social trust\. Respondents are grouped by country, survey wave, sex, age, education, employment, and marital status, yielding 687 demographic cells for three questions and 639 for political ideology\. Each cell is represented by an interview\-style demographic prompt\. Responses are elicited either from renormalized next\-token probabilities \(NTP\) or from sampled full answers \(FA\)\. Correspondence between survey and model distributions is measured using normalized Earth Mover’s Distance:

nEMD⁡\(p,q\)=1K−1​∑k=1K−1\|∑j=1k\(pj−qj\)\|,\\operatorname\{nEMD\}\(p,q\)=\\frac\{1\}\{K\-1\}\\sum\_\{k=1\}^\{K\-1\}\\left\|\\sum\_\{j=1\}^\{k\}\(p\_\{j\}\-q\_\{j\}\)\\right\|,\(1\)whereppandqqare distributions overKKordered response options, so thatnEMD∈\[0,1\]\\operatorname\{nEMD\}\\in\[0,1\]\. This metric supplies the distance used throughout Section[4](https://arxiv.org/html/2609.36253#S4)\.

Before applying our framework, we check that our independent implementation reproduces the original results\. Passing the archived generations released by[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)through our pipeline reproduces every full\-precision checkpoint reported for Mixtral\-8x7B, along with the published nEMD distributions, regression analyses, and multidimensional\-scaling results\. We also regenerate Mixtral\-8x7B responses from the public weights using the original prompts\. This fresh run is equivalent to the archived one under NTP but not fully under FA; the two runs differ in quantization and inference backend as well as sampling\. We retain both as reference series, since the archived run preserves comparability with the published results, and the fresh run provides a reference under our current inference setup\. Appendix[A](https://arxiv.org/html/2609.36253#A1)reports the reproduction and equivalence tests in full\.

## 4The Population Fidelity Framework

The analyses of[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)diagnose machine bias at the level of an entire model run\. We decompose population fidelity into quantities that can be computed for a complete population or any sufficiently large subset of its subpopulations\. The framework distinguishes whether a model approximates particular groups, reproduces the observed amount of variation between groups, and preserves the structure of those differences\.

### 4\.1Unit of Analysis and Notation

Consider a survey question withKKordered response options and a model evaluated onnnretained subpopulations, which we call*cells*\. Leti=1,…,ni=1,\\ldots,nindex the cells, withsis\_\{i\}denoting the empirical survey\-response distribution andmim\_\{i\}the corresponding model\-response distribution\. Both are probability vectors over the same response categories\. Distances are calculated using Equation[1](https://arxiv.org/html/2609.36253#S3.E1)\.

### 4\.2Population Fidelity Dimensions

Accuracy\.Average cell\-level error isE=1n​∑i=1nnEMD⁡\(si,mi\)\.E=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\operatorname\{nEMD\}\(s\_\{i\},m\_\{i\}\)\.We define the accuracy score asSacc=1−ES\_\{\\mathrm\{acc\}\}=1\-E\. It measures how closely the model approximates each group’s observed response distribution and corresponds to the per\-cell distances used in the original Machine Bias study\.

Adaptability\.Accuracy does not indicate whether the model reproduces variation between social groups\. To measure this, we first calculate the median pairwise distances among model and survey cells,DLLM=mediani<j⁡\{nEMD⁡\(mi,mj\)\}D\_\{\\mathrm\{LLM\}\}=\\operatorname\{median\}\_\{i<j\}\\\{\\operatorname\{nEMD\}\(m\_\{i\},m\_\{j\}\)\\\}andDWVS=mediani<j⁡\{nEMD⁡\(si,sj\)\}D\_\{\\mathrm\{WVS\}\}=\\operatorname\{median\}\_\{i<j\}\\\{\\operatorname\{nEMD\}\(s\_\{i\},s\_\{j\}\)\\\}\. We define the adaptability ratio asA=DLLMDWVS\.A=\\frac\{D\_\{\\mathrm\{LLM\}\}\}\{D\_\{\\mathrm\{WVS\}\}\}\.This measures whether the model produces as much between\-group variation as the survey\. A value ofA=1A=1indicates equal dispersion, while values below and above one indicate compression and amplification, respectively\. Because both compression and amplification depart from the survey, we define the symmetric adaptability score asSadapt=min⁡\(A,A−1\)S\_\{\\mathrm\{adapt\}\}=\\min\(A,A^\{\-1\}\)forA\>0A\>0, and setSadapt=0S\_\{\\mathrm\{adapt\}\}=0whenA=0A=0\. Halving and doubling the survey’s dispersion therefore receive the same score\.

Structure\.A model can reproduce the overall amount of variation while assigning it to the wrong groups\. We compare the complete vectors of pairwise distances usingρ=corr⁡\(\{nEMD⁡\(si,sj\)\}i<j,\{nEMD⁡\(mi,mj\)\}i<j\),\\rho=\\operatorname\{corr\}\\left\(\\left\\\{\\operatorname\{nEMD\}\(s\_\{i\},s\_\{j\}\)\\right\\\}\_\{i<j\},\\left\\\{\\operatorname\{nEMD\}\(m\_\{i\},m\_\{j\}\)\\right\\\}\_\{i<j\}\\right\),wherecorr\\operatorname\{corr\}is Spearman’s rank correlation\. A value near one indicates that the same pairs of groups are relatively similar or different in the model and survey responses\. A value near zero indicates little correspondence between the two structures\. The combinationA\>1A\>1andρ≈0\\rho\\approx 0, which we call*over\-steerability without social fidelity*, occurs when demographic conditioning produces strong differences that do not correspond to those observed among human groups\. The pairwise\-distance vectors contain\(n2\)\\binom\{n\}\{2\}entries that share cells and are not independent\. We therefore treatρ\\rhoas a descriptive effect size rather than interpreting its naive significance test\([Mantel, 1967](https://arxiv.org/html/2609.36253#bib.bib23)\)\. The structure score isSstruct=max⁡\(0,ρ\)S\_\{\\mathrm\{struct\}\}=\\max\(0,\\rho\), so a nonpositive association indicates no recovered population structure\. We retain the untransformedρ\\rhofor diagnostic analyses, particularly when interpreting values near zero\.

### 4\.3Population Fidelity Score and Center Alignment

Population Fidelity Score \(PFS\)is the geometric mean of accuracy, adaptability, and structure:PFS=\(Sacc⋅Sadapt⋅Sstruct\)1/3\.\\centering\\mathrm\{PFS\}=\\left\(S\_\{\\mathrm\{acc\}\}\\cdot S\_\{\\mathrm\{adapt\}\}\\cdot S\_\{\\mathrm\{struct\}\}\\right\)^\{1/3\}\.\\@add@centeringWe use the geometric mean because all three dimensions are required for population fidelity\. A synthetic population must approximate the responses of specific groups, reproduce the observed amount of between\-group variation, and assign that variation to the corresponding groups\. An arithmetic mean lets strong dimensions offset failure on another\. The geometric mean instead makes weak components constrain the overall score, as in multidimensional indices such as the Human Development Index\([Klugman et al\., 2011](https://arxiv.org/html/2609.36253#bib.bib19)\)\.

Center alignment \(Sc​e​n​t​e​rS\_\{center\}\)\.Center alignment is reported alongside the PFS but is not included in it\. For a set of retained cellsII, define the survey center ass¯I=1\|I\|​∑i∈Isi,\\bar\{s\}\_\{I\}=\\frac\{1\}\{\|I\|\}\\sum\_\{i\\in I\}s\_\{i\},and the model center asm¯I=1\|I\|​∑i∈Imi\.\\bar\{m\}\_\{I\}=\\frac\{1\}\{\|I\|\}\\sum\_\{i\\in I\}m\_\{i\}\.The distance between them isCI=nEMD⁡\(s¯I,m¯I\),C\_\{I\}=\\mathrm\{nEMD\}\(\\bar\{s\}\_\{I\},\\bar\{m\}\_\{I\}\),andScenter=1−CI\.S\_\{\\mathrm\{center\}\}=1\-C\_\{I\}\.We report center alignment over all retained cells and, for cultural fine\-tuning analyses, over retained cells in the target country\. Because the centers weight retained cells equally, center alignment measures agreement between the model and survey centers rather than a respondent\-weighted population mean\. A model can match this center while remaining nearly invariant across demographic groups\. We therefore report center alignment separately from the PFS since a well\-aligned center does not compensate for missing variation or structure\.

### 4\.4Population Fidelity within Demographic Subgroups

Each quantity can be recomputed within a subset of cells\. For a demographic groupggwith cell setℐg\\mathcal\{I\}\_\{g\}, we calculate accuracy, adaptability, structure, center alignment, and PFS using only cells inℐg\\mathcal\{I\}\_\{g\}\. We consider levels within seven demographic categories: country, survey wave, sex, age, education, employment, and marital status\. For each subgroup, we recompute all metrics using only the cells belonging to that subgroup, rather than averaging population\-level scores\. This allows the framework to identify population segments for which a model or intervention is more or less faithful\.

## 5Experimental Design

### 5\.1Models and Conditions

Appendix[B\.1](https://arxiv.org/html/2609.36253#A2.SS1)provides an overview of the experimental pipeline\. We evaluate six open\-weight models spanning approximately 2B–31B parameters: Gemma\-4\-31B\-it, Gemma\-4\-E4B\-it, Qwen3\-VL\-8B\-Thinking, Qwen3\-VL\-2B\-Thinking, Llama\-3\.2\-3B, and Muse\-Glimmer\-30B\. Each is evaluated as released and after cultural fine\-tuning toward Germany and Mexico\. We also evaluate OpenAI’s*GPT\-5\.6 Terra*\(Terra\), a proprietary model accessed through its API\. Terra is evaluated as released under FA elicitation only because its endpoint does not provide usable NTP at nonzero reasoning effort\. Mixtral\-8x7B is included through two reproduction references: archived generations from the original Machine Bias study and responses regenerated from the public model weights\.

The design contains 84 model–condition–question series: 80 from 20 conditions evaluated on four questions, plus four Terra series\. The non\-Terra series are evaluated under both NTP and FA, yielding 160 run–mode combinations; Terra is FA\-only, for a total of 164 combinations\.

### 5\.2Cultural Fine\-Tuning

In this study, we follow the approach taken by CultureLLM\([Li et al\., 2024](https://arxiv.org/html/2609.36253#bib.bib20)\)\. For each target culture, the training data combine CultureLLM’s selected WVS seed questions with semantically augmented paraphrases and culture\-specific target answers derived from aggregate survey responses\.

We use CultureLLM’s German condition and construct a Mexican condition\. CultureLLM’s original Spanish partition pools respondents from Mexico and Argentina, whereas our reference population is Mexico\. We therefore reconstruct the training records using only Mexican WVS responses\. We denote the resulting German and Mexican fine\-tuned conditions byℱde\\mathcal\{F\}\_\{\\text\{de\}\}andℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}, respectively\.

Each variant is implemented as a low\-rank update to its corresponding as\-released model using LoRA with rankr=8r=8and scaling parameterα=16\\alpha=16\([Hu et al\., 2021](https://arxiv.org/html/2609.36253#bib.bib15)\)\. We use QLoRA for the two largest models\([Dettmers et al\., 2023](https://arxiv.org/html/2609.36253#bib.bib11)\)\. The German and Mexican variants of a model use the same training and inference configuration\. None of the four evaluation items—happiness, political ideology, religious attendance, or social trust—appears in the fine\-tuning data\.

### 5\.3Elicitation and Evaluation

We use the interview\-style prompts \(Section[3](https://arxiv.org/html/2609.36253#S3)\) under both elicitation modes, NTP and FA; FA responses are sampled at temperature 0\.7 and parsed against the offered options\. A 32\-prompt capacity probe precedes each run and flags an NTP run as non\-informative if its mean probability mass on valid answer tokens is below0\.100\.10; none is flagged \(Appendix[C](https://arxiv.org/html/2609.36253#A3)\)\. A cell enters the analysis only with at least 20 valid FA responses\. NTP is deterministic under our decoding configuration and is run once\. Because FA samples stochastically, we run each of the 18 locally evaluated conditions three times\. For consistency with conditions available for only one FA run, primary results use the first FA run; figures show three\-run means and 95% CIs where repeats are available \(Appendix[E\.1](https://arxiv.org/html/2609.36253#A5.SS1)\)\.

We compute every metric at three scopes: population\-wide, using all retained cells; target\-country, using cells from the fine\-tuning target country \(German forℱde\\mathcal\{F\}\{\{\}\_\{\\text\{de\}\}\}, Mexican forℱmx\\mathcal\{F\}\{\{\}\_\{\\text\{mx\}\}\}\); and subgroup\-specific, recomputing the metric within each level of country, survey wave, sex, age, education, employment, and marital status \(Section[4\.4](https://arxiv.org/html/2609.36253#S4.SS4)\)\. We report fine\-tuning effects at both population\-wide and target\-country scopes\.

Each fine\-tuned condition is compared only with its corresponding as\-released model\. Under FA elicitation, the set of retained cells can differ across conditions because a cell must contain at least 20 valid responses for that condition\. To ensure that paired comparisons are based on the same population, each fine\-tuned condition and its as\-released baseline are scored on the intersection of the cells retained by both\. Thus, each paired difference reflects changes in model responses on the same cells, rather than differences in cell composition\. With six models, four questions, and two elicitation modes, this yields 48 paired comparisons per cultural target\.

All primary analyses are conducted separately by question; averages across the four questions are reported only as descriptive summaries\. Appendix[B\.2](https://arxiv.org/html/2609.36253#A2.SS2)details metric implementation and retention; Appendix[C](https://arxiv.org/html/2609.36253#A3)provides details on experimental setup, fine\-tuning, response coverage, and prompts\.

## 6Results

### 6\.1LLMs Approximate the Survey Center but Miss Group Structure

Models often achieve high cell\-level accuracy but rarely recover the structure of differences among cells\. To contextualize accuracy, we use a survey\-center baseline that ignores demographic information \(Appendix[D\.1](https://arxiv.org/html/2609.36253#A4.SS1)\), predicting the equal\-cell survey centers¯I\\bar\{s\}\_\{I\}for every cell\. This baseline attainsSacc∈\[0\.848,0\.915\]S\_\{\\mathrm\{acc\}\}\\in\[0\.848,0\.915\]across questions and matches or exceeds model accuracy in 157 of 164 run–mode combinations; only seven combinations outperform it\. Thus, much of the observed accuracy reflects proximity to the survey center rather than recovery of demographic differences\.

Across the 164 combinations, medianSacc=0\.832S\_\{\\mathrm\{acc\}\}=0\.832whereas medianSstruct=0\.060S\_\{\\mathrm\{struct\}\}=0\.060\(Appendix[E\.3](https://arxiv.org/html/2609.36253#A5.SS3), Table[13](https://arxiv.org/html/2609.36253#A5.T13), default row\)\.SstructS\_\{\\mathrm\{struct\}\}is the limiting component in 159 combinations,SadaptS\_\{\\mathrm\{adapt\}\}in five, andSaccS\_\{\\mathrm\{acc\}\}in none\. The survey\-center baseline itself hasScenter=1S\_\{\\mathrm\{center\}\}=1butA=0A=0, and thereforePFS=0\\mathrm\{PFS\}=0, illustrating why center agreement alone is insufficient\. Another 30 combinations havePFS=0\\mathrm\{PFS\}=0becauseρ≤0\\rho\\leq 0; their narrow range,ρ∈\[−0\.026,0\]\\rho\\in\[\-0\.026,0\], indicates absent rather than systematically reversed population structure\.

Figure[2\(a\)](https://arxiv.org/html/2609.36253#S6.F2.sf1)shows this profile for happiness under FA elicitation: accuracy is high for every condition, whereas structure is the weakest component for almost all\. The other questions show the same pattern \(Appendix[D\.5](https://arxiv.org/html/2609.36253#A4.SS5)\)\. This parallels the machine bias pattern of[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6), in which errors are largely explained by each cell’s distance from the model’s average answer, and a model that barely varies is accurate only insofar as its center sits near the survey cells\. Population Fidelity complements this diagnosis by showing that such accuracy can coexist with weak recovery of the observed structure of between\-group differences\.

\(a\)Component profile\.\(b\)Fine\-tuning shifts\.
Figure 2:Population Fidelity for happiness under FA elicitation\.\(a\)Components, PFS, and center alignment for the indicated cell sets\.\(b\)Fine\-tuned minus as\-released changes on target\-country cells\. Markers show three\-run FA means using each run’s paired support; Table[10](https://arxiv.org/html/2609.36253#A5.T10), Appendix[E\.1](https://arxiv.org/html/2609.36253#A5.SS1), uses the stricter all\-runs common\-cell intersection\. All\-cell results are in Figure[13](https://arxiv.org/html/2609.36253#A4.F13), Appendix[D\.5](https://arxiv.org/html/2609.36253#A4.SS5)\.
### 6\.2More Demographic Variation Does Not Imply Better Group Structure

[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)identify low adaptability as the core of machine bias\. Our ratioAAformalizes their comparison of median pairwise distances among model and survey cells \(e\.g\.,0\.033/0\.115≈0\.290\.033/0\.115\\approx 0\.29for Mixtral\-8x7B on happiness\); structure asks whether the variation a model does produce falls between the right groups\. Among the 52 as\-released run–mode combinations excluding the Mixtral reference series, 43 compress variation relative to the survey \(A<1A<1\), whereas nine amplify it \(A\>1A\>1\)\. Compression occurs for every combination on happiness and political ideology, while amplification appears only for religious attendance and social trust \(Table[7](https://arxiv.org/html/2609.36253#A4.T7), Appendix[D\.3](https://arxiv.org/html/2609.36253#A4.SS3)\)\. Yet greater variation rarely corresponds to better group structure: only one of the nine amplifying combinations reachesρ\>0\.1\\rho\>0\.1\(Figure[12](https://arxiv.org/html/2609.36253#A4.F12), Appendix[D\.5](https://arxiv.org/html/2609.36253#A4.SS5), plotsAAagainstρ\\rhounder NTP\)\. Thus, models can respond differently across demographic profiles without reproducing which groups are relatively similar or different in the survey\. Adaptability and structure therefore capture distinct requirements of population fidelity: matching the amount of variation is not sufficient if that variation is distributed across the wrong groups\.

### 6\.3CultureLLM\-Style Fine\-Tuning Shows No Consistent Fidelity Gains

In the primary analysis, CultureLLM\-style fine\-tuning does not improve PFS on average in either scope\. Here,Δ​PFS\\Delta\\mathrm\{PFS\}denotes fine\-tuned minus as\-released PFS, so positive values indicate improved population fidelity\. Population\-wide, effects are mixed:ℱde\\mathcal\{F\}\_\{\\text\{de\}\}raises PFS in 21 of 48 comparisons and lowers it in 23 \(meanΔ​PFS=−0\.024\\Delta\\mathrm\{PFS\}=\-0\.024\), whileℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}raises it in 19 and lowers it in 23 \(mean−0\.014\-0\.014\)\. Target\-country means are also negative:−0\.021\-0\.021in Germany and−0\.010\-0\.010in Mexico\. Thus, the aggregate result reflects heterogeneous effects rather than a consistent benefit from fine\-tuning\. Model\- and question\-specific results are reported in Appendix[D\.2](https://arxiv.org/html/2609.36253#A4.SS2)\.

The repeated FA runs support the same conclusion but show greater sampling sensitivity for PFS than center alignment\. Across runs, meanΔ​PFS\\Delta\\mathrm\{PFS\}on all retained cells and meanΔ​Scenter\\Delta S\_\{\\mathrm\{center\}\}in both scopes vary by at most0\.0050\.005\. Within target countries, pooled 95% CI half\-widths for individual\-comparisonΔ​PFS\\Delta\\mathrm\{PFS\}are0\.0730\.073in Germany and0\.1100\.110in Mexico, versus0\.0050\.005and0\.0030\.003forΔ​Scenter\\Delta S\_\{\\mathrm\{center\}\}, respectively \(Appendix[E\.1](https://arxiv.org/html/2609.36253#A5.SS1)\)\.

Changes in center alignment do not necessarily track changes in PFS\. Figure[2\(b\)](https://arxiv.org/html/2609.36253#S6.F2.sf2)shows this for happiness within Germany: Gemma\-4\-31B\-it’sℱde\\mathcal\{F\}\_\{\\text\{de\}\}increases target\-country PFS by0\.3710\.371while barely changing center alignment \(Δ​Scenter=−0\.004\\Delta S\_\{\\mathrm\{center\}\}=\-0\.004\), whereas Qwen3\-VL\-8B’sℱde\\mathcal\{F\}\_\{\\text\{de\}\}decreases both \(Δ​PFS=−0\.015\\Delta\\mathrm\{PFS\}=\-0\.015,Δ​Scenter=−0\.164\\Delta S\_\{\\mathrm\{center\}\}=\-0\.164\)\. The same appears in the cross\-question FA summary\. Across models and questions,ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}moves responses toward the Mexican survey center \(Δ​Scenter=\+0\.037\\Delta S\_\{\\mathrm\{center\}\}=\+0\.037\) while target\-country PFS changes little \(Δ​PFS=−0\.002\\Delta\\mathrm\{PFS\}=\-0\.002\);ℱde\\mathcal\{F\}\_\{\\text\{de\}\}moves responses away from the German center \(Δ​Scenter=−0\.087\\Delta S\_\{\\mathrm\{center\}\}=\-0\.087\) while target\-country PFS changes by−0\.017\-0\.017\.

The components clarify this pattern\. Within the target countries, fine\-tuning reduces adaptability: medianSadaptS\_\{\\mathrm\{adapt\}\}falls from0\.7540\.754to0\.4960\.496in Germany and from0\.6160\.616to0\.4210\.421in Mexico\. Structure changes little \(Δ​Sstruct=0\.000\\Delta S\_\{\\mathrm\{struct\}\}=0\.000in Germany and−0\.004\-0\.004in Mexico\) and remains the limiting PFS component in 44 of 48ℱde\\mathcal\{F\}\_\{\\text\{de\}\}and 47 of 48ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}comparisons\. Thus, fine\-tuning can shift average responses and alter between\-group variation without recovering the observed pattern of group differences\. PFS captures these distinctions across fine\-tuning objectives\. Appendix[F](https://arxiv.org/html/2609.36253#A6)illustrates the same diagnostics for CultureLLM\-style, distribution\-matched, and subgroup\-matched fine\-tuning on a common backbone\. Because these conditions differ in training data, population targets, and evaluation\-item overlap, the comparison is diagnostic rather than controlled\. On Qwen3\-VL\-2B\-Thinking, the distribution\-matched condition retains higher adaptability and structure than the CultureLLM\-style variants, whereas the subgroup\-matched condition remains structure\-limited\.

### 6\.4Center Alignment Can Mask Poor Fidelity within Demographic Subgroups

Within demographic levels, subgroup centers are generally well matched while PFS varies widely, so a matched center is weak evidence of fidelity inside the group\. For happiness under FA elicitation, medianScenterS\_\{\\mathrm\{center\}\}across demographic levels ranges from0\.8480\.848to0\.9280\.928\(Figure[3](https://arxiv.org/html/2609.36253#S6.F3)\)\. Across model conditions, PFS andScenterS\_\{\\mathrm\{center\}\}are only moderately correlated within demographic levels, with mean Spearman correlations ranging from0\.2630\.263to0\.4250\.425\(Appendix[D\.4](https://arxiv.org/html/2609.36253#A4.SS4)\)\.

Figure 3:PFS and center alignment by demographic level for happiness under FA elicitation\. Boxes summarize 19 model conditions\. Results for all questions: Figure[10](https://arxiv.org/html/2609.36253#A4.F10)\.The imbalance persists across demographic levels, and fine\-tuning does not repair it\. Across the 4,440 subgroup\-level scores \(148 run–mode combinations by 30 levels, Mixtral references excluded\),SstructS\_\{\\mathrm\{struct\}\}is limiting in 4,097 andSadaptS\_\{\\mathrm\{adapt\}\}in 343, whileSaccS\_\{\\mathrm\{acc\}\}never is; PFS varies much more across age and employment than across sex: the mean within\-condition range is0\.2870\.287for age and0\.2700\.270for employment, versus0\.0540\.054for sex, with these between\-level differences driven more by structure than by adaptability \(Table[8](https://arxiv.org/html/2609.36253#A4.T8), Appendix[D\.4](https://arxiv.org/html/2609.36253#A4.SS4)\)\. German and Mexican fine\-tuning improve PFS in only42\.5%42\.5\\%and40\.8%40\.8\\%of subgroup comparisons, respectively; within Germany and Mexico, PFS rises in only 18 of 48 country\-level comparisons for each target \(Figure[9](https://arxiv.org/html/2609.36253#A4.F9), Appendix[D\.4](https://arxiv.org/html/2609.36253#A4.SS4)\)\.

### 6\.5Robustness Across Elicitation, Sampling, and Metric Specifications

The main patterns are broadly consistent across elicitation modes\. Among the 20 conditions evaluated under both NTP and FA, PFS rank correlations range from0\.6950\.695for political ideology to0\.8800\.880for religious attendance\. Agreement is generally higher forSaccS\_\{\\mathrm\{acc\}\}andScenterS\_\{\\mathrm\{center\}\}, whileSadaptS\_\{\\mathrm\{adapt\}\}is more sensitive to elicitation, with correlations from0\.5020\.502to0\.9130\.913\(Table[12](https://arxiv.org/html/2609.36253#A5.T12), Appendix[E\.2](https://arxiv.org/html/2609.36253#A5.SS2)\)\. Fine\-tuning effects are also directionally similar: NTP and FA agree on the sign ofΔ​PFS\\Delta\\mathrm\{PFS\}in 38 of 48 comparisons, or 35 of 41 after excluding cases with zero change under either mode \(Appendix[E\.2](https://arxiv.org/html/2609.36253#A5.SS2)\)\.

The conclusions are also stable across alternative metric specifications\. Appendix[E\.3](https://arxiv.org/html/2609.36253#A5.SS3)varies the dispersion statistic, adaptability transformation, structure correlation and clipping, aggregation rule, and cell weighting\. Across these cases,SstructS\_\{\\mathrm\{struct\}\}remains the limiting component in at least 156 of 164 combinations, and neither CultureLLM\-style target improves population\-wide PFS on average under any specification\. Replacing the geometric mean with an arithmetic mean changes PFS more because it allows stronger components to offset weak structure\. Mean PFS rises from0\.2430\.243to0\.4510\.451and no longer reaches zero, illustrating why the geometric mean is better aligned with our requirement that all three dimensions be jointly satisfied\.

## 7Discussion, Limitations, and Conclusion

Overall, our work extends existing efforts to understand the extent to which LLMs are capable of simulating known patterns of human attitudes and beliefs\. While early work featured forms of social or demographic bias, in which one group’s attitudes might be systematically favored over others, more recent research has found that LLMs themselves exhibit their own characteristic biases\. Most notably, they tend to reduce variation across demographic groups\. This paper builds upon such work to formulate a more general framework for evaluating thepopulation fidelityof LLM\-generated survey response patterns\.

In general, we find that LLM\-generated responses do approximate the centers of human survey responses\. However, they do a poor job of recovering differences among groups, a task which is often central to social and cultural research\. In other words, weak fidelity to the typical response patterns we observe among human survey respondents typically occurs not simply from recovering too little demographic variation, but from failing to reproduce which groups respond in ways that are relatively similar or different to each other\. Our Population Fidelity metric makes this distinction explicit by separating fidelity into three components: group\-level accuracy, the amount of between\-group variation, and its structure\. In our experiments, structure is usually the limiting dimension, also under the extra adaptation objectives evaluated in Appendix[F](https://arxiv.org/html/2609.36253#A6)\.

Our analysis is limited in a number of ways, which future work can seek to address\. First, the scope of the data sources could be expanded\. We have evaluated just one dataset, the WVS, and from that focused on a limited subset: three waves, five countries, four attitudes, and a set of relatively coarse demographic cells\. Second, even though respondents are international, we have used English prompts and fine\-tuning data\. Likewise, our examination of fine\-tuning approaches evaluated six models and two cultural targets, whereas the two additional fine\-tuning methods we pursued \(SimLLCultureDist and SubPOP\) were tested on only one backbone\. While these experiments demonstrate the applicability of PFS across adaptation objectives, testing across more objectives and sources and under additional conditions could help to understand the promise and limits of the framework\. Finally, the method used to elicit responses also matters\. Whether a model appears to compress or amplify differences between demographic groups depends partly on whether responses are derived from next\-token probabilities \(NTP\) or sampled as full answers \(FA\)\. Adaptability is more sensitive to this choice than the other components of Population Fidelity, so compression and amplification should generally be compared within the same elicitation mode\. Further tests that systematically vary elicitation methods would help to understand these dynamics more fully\.

Despite these limitations, this paper has established that Population Fidelity provides a common framework for asking not only whether an intervention improves alignment with a target population, but which aspects of population representation it does or does not recover\. Across the evaluations, models rarely recover the structure of group differences observed in human populations; evaluations based on agreement with the survey center tend to obscure such misrepresentations\. Additional experiments have illustrated how the same framework can distinguish the effects of different fine\-tuning objectives\. Future work should extend these tests to broader populations, languages, surveys, and model families; scale distribution\-based methods beyond a single backbone; and examine richer representations of cultural and individual values, including datasets designed to capture heterogeneous and potentially conflicting values within the same population, as well as established frameworks that represent systems of values rather than isolated survey attitudes\. More generally, these extensions would allow researchers to ask not simply whether a model resembles a population on average, but whether it reproduces the ways in which that population is internally differentiated\.

### AI Use Statement

Generative AI tools were used during the preparation of this work to help improve the clarity and organization of the manuscript, to support brainstorming and the organization of ideas, and to assist with software development and debugging\. AI\-assisted text and code were reviewed and, where relevant, revised and tested by the authors\. The authors take full responsibility for the final manuscript, analyses, code, and reported results\.

### Ethics Statement

This work evaluates the extent to which LLM\-generated survey responses represent patterns observed in human survey data\. The analysis uses the World Values Survey through the demographic cells and survey items defined by[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)and does not involve the recruitment of new human participants\. Because demographic conditioning and population simulation can reproduce or amplify stereotypes, results for demographic groups should not be interpreted as claims about individual members of those groups\. Population Fidelity is intended as an evaluation framework for identifying discrepancies between model\-generated and observed population patterns, not as evidence that demographic categories fully characterize the attitudes or behavior of the populations they describe\.

### Reproducibility Statement

We provide the information needed to reproduce the experiments in the main paper and appendices, including the survey and model evaluation procedure, Population Fidelity metrics, model and fine\-tuning configurations, elicitation settings, retention criteria, reproduction checks, and sensitivity analyses\. The accompanying repositories contain the code, prompts, configuration files, run manifests, and scripts used to produce the reported results and figures\. One repository contains the evaluation of the as\-released models and the Machine Bias reproduction,111[https://github\.com/CriticalMaking/LLM\-population\-fidelity](https://github.com/CriticalMaking/LLM-population-fidelity)while the second contains the cultural fine\-tuning experiments and their evaluation\.222[https://github\.com/neemiasbsilva/CultureFinetuneSteerMLLM](https://github.com/neemiasbsilva/CultureFinetuneSteerMLLM)Appendix[A](https://arxiv.org/html/2609.36253#A1)documents the Machine Bias reproduction; Appendix[B](https://arxiv.org/html/2609.36253#A2)describes the Population Fidelity framework, metric implementation, and toy example; Appendix[C](https://arxiv.org/html/2609.36253#A3)provides the experimental configuration and prompts; Appendix[D](https://arxiv.org/html/2609.36253#A4)reports supplementary results; Appendix[E](https://arxiv.org/html/2609.36253#A5)presents robustness and sensitivity analyses; and Appendix[F](https://arxiv.org/html/2609.36253#A6)evaluates additional adaptation objectives\.

#### Acknowledgments

This study was supported in part by the National Council for Scientific and Technological Development \- CNPq \(processes 441444/2023\-7, 403646/2026\-0, and 444724/2024\-9\) and INCT TILD\-IAR \(proc\. 408490/2024\-1\)\. This research was also supported in part by a CIFAR AI Catalyst Grant, “Towards Socially Grounded AI Safety: Integrating Causal and Institutional Reasoning in Language Models” \(PI: Matt Ratto; Co\-PI: Zhijing Jin, University of Toronto\)\.

## References

- Adilazuarda et al\. \(2025\)Farid Adilazuarda, Chen Cecilia Liu, Iryna Gurevych, and Alham Fikri Aji\.From surveys to narratives: Rethinking cultural value adaptation in LLMs\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pp\. 18063–18090, 2025\.
- Argyle et al\. \(2023\)Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate\.Out of one, many: Using language models to simulate human samples\.*Political Analysis*, 31\(3\):337–351, 2023\.
- Bai et al\. \(2025\)Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al\.Qwen3\-VL technical report\.*arXiv preprint arXiv:2511\.21631*, 2025\.
- Bail \(2024\)Christopher A\. Bail\.Can Generative AI improve social science?*Proceedings of the National Academy of Sciences*, 121\(21\):e2314021121, May 2024\.doi:10\.1073/pnas\.2314021121\.
- Bisbee et al\. \(2024\)James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson\.Synthetic replacements for human survey data? The perils of large language models\.*Political Analysis*, 32\(4\):401–416, 2024\.
- Boelaert et al\. \(2025\)Julien Boelaert, Samuel Coavoux, Étienne Ollion, Ivaylo Petev, and Patrick Präg\.Machine bias\. How do generative language models answer opinion polls?*Sociological Methods & Research*, 54\(3\):1156–1196, 2025\.
- Cao et al\. \(2025\)Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich\.Specializing large language models to simulate survey response distributions for global populations\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 3141–3154, 2025\.
- Cheng et al\. \(2023\)Myra Cheng, Esin Durmus, and Dan Jurafsky\.Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models\.In Anna Rogers, Jordan Boyd\-Graber, and Naoaki Okazaki \(eds\.\),*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1504–1532, Toronto, Canada, July 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.acl\-long\.84\.
- Davidson & Karell \(2025\)Thomas Davidson and Daniel Karell\.Integrating Generative Artificial Intelligence into Social Science Research: Measurement, Prompting, and Simulation\.*Sociological Methods & Research*, 54\(3\):775–793, August 2025\.ISSN 0049\-1241\.doi:10\.1177/00491241251339184\.
- Deshpande et al\. \(2023\)Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan\.Toxicity in ChatGPT: Analyzing persona\-assigned language models\.In Houda Bouamor, Juan Pino, and Kalika Bali \(eds\.\),*Findings of the Association for Computational Linguistics: EMNLP 2023*, pp\. 1236–1270, Singapore, December 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.findings\-emnlp\.88\.
- Dettmers et al\. \(2023\)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.QLoRA: Efficient finetuning of quantized LLMs\.*Advances in neural information processing systems*, 36:10088–10115, 2023\.
- Dillion et al\. \(2023\)Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray\.Can AI language models replace human participants?*Trends in Cognitive Sciences*, 27\(7\):597–600, July 2023\.ISSN 13646613\.doi:10\.1016/j\.tics\.2023\.04\.008\.
- Gallegos et al\. \(2024\)Isabel O\. Gallegos, Ryan A\. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K\. Ahmed\.Bias and Fairness in Large Language Models: A Survey\.*Computational Linguistics*, 50\(3\):1097–1179, September 2024\.ISSN 0891\-2017\.doi:10\.1162/coli\_a\_00524\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Hu et al\. \(2021\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*, 2021\.
- Karetnikov et al\. \(2026\)Nikita Karetnikov, Iyad Rahwan, and Davor Svetinovic\.Large language models as human proxies\.*Nature Computational Science*, pp\. 1–10, September 2026\.ISSN 2662\-8457\.doi:10\.1038/s43588\-026\-01060\-3\.
- Keeter et al\. \(2017\)Scott Keeter, Nick Hatley, Courtney Kennedy, and Arnold Lau\.What Low Response Rates Mean for Telephone Surveys\.Technical report, Pew Research Center, 2017\.
- Kim & Lee \(2026\)Junsol Kim and Byungkyu Lee\.AI\-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction\.*arXiv preprint arXiv:2305\.09620*, 2026\.doi:10\.48550/arXiv\.2305\.09620\.
- Klugman et al\. \(2011\)Jeni Klugman, Francisco Rodríguez, and Hyung\-Jin Choi\.The HDI 2010: New controversies, old critiques\.*The Journal of Economic Inequality*, 9\(2\):249–288, 2011\.
- Li et al\. \(2024\)Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie\.CultureLLM: Incorporating cultural differences into large language models\.*Advances in Neural Information Processing Systems*, 37:84799–84838, 2024\.
- Liu et al\. \(2024\)Andy Liu, Mona Diab, and Daniel Fried\.Evaluating Large Language Model Biases in Persona\-Steered Generation\.In Lun\-Wei Ku, Andre Martins, and Vivek Srikumar \(eds\.\),*Findings of the Association for Computational Linguistics: ACL 2024*, pp\. 9832–9850, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-acl\.586\.
- Lukk \(2026\)Martin Lukk\.FairFund\-Bench: Evaluating Distributive Bias in LLM Resource Allocation\.*arXiv preprint arXiv:2607\.28934*, 2026\.doi:10\.48550/arXiv\.2607\.28934\.
- Mantel \(1967\)Nathan Mantel\.The detection of disease clustering and a generalized regression approach\.*Cancer Research*, 27\(2\_Part\_1\):209–220, 1967\.
- Meta Superintelligence Labs \(2026\)Meta Superintelligence Labs\.Introducing Muse Glimmer: An open agentic model that runs on your device\.Meta AI Research blog, 2026\.URL[https://research\.meta\.ai/blog/introducing\-muse\-glimmer\-open\-agentic\-model](https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model)\.Released August 10, 2026\.
- Santurkar et al\. \(2023\)Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto\.Whose opinions do language models reflect?In*International conference on machine learning*, pp\. 29971–30004\. PMLR, 2023\.
- Suh et al\. \(2025\)Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang\.Language model fine\-tuning on scaled survey data for predicting distributions of public opinions\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 21147–21170, 2025\.
- Team et al\. \(2026\)Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al\.Gemma 4 technical report\.*arXiv preprint arXiv:2607\.02770*, 2026\.
- Wan et al\. \(2023\)Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai\-Wei Chang\.Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems\.In Houda Bouamor, Juan Pino, and Kalika Bali \(eds\.\),*Findings of the Association for Computational Linguistics: EMNLP 2023*, pp\. 9677–9705, Singapore, December 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.findings\-emnlp\.648\.
- Wang et al\. \(2025\)Angelina Wang, Jamie Morgenstern, and John P Dickerson\.Large language models that replace human participants can harmfully misportray and flatten identity groups\.*Nature Machine Intelligence*, 7\(3\):400–411, 2025\.
- Wightman et al\. \(2026\)Maria\-Louisa Wightman, Guillaume Bied, and Tijl De Bie\.People are not just their countries\. Disentangling social determinants of LLM value alignment across Europe\.*arXiv preprint arXiv:2608\.07367*, 2026\.
- Williams et al\. \(2026\)Tristan Williams, Franziska Weeber, Sebastian Padó, and Alan Akbik\.Beyond marginal distributions: A framework to evaluate the representativeness of demographic\-aligned LLMs\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pp\. 4806–4826, 2026\.
- Zhang et al\. \(2025\)Simone Zhang, Janet Xu, and AJ Alvero\.Generative AI Meets Open\-Ended Survey Responses: Research Participant Use of AI and Homogenization\.*Sociological Methods & Research*, 54\(3\):1197–1242, August 2025\.ISSN 0049\-1241\.doi:10\.1177/00491241251327130\.

## Appendix

## Appendix AMachine Bias Design and Reproduction Details

This appendix describes the original Machine Bias design, our two\-stage reproduction, and the equivalence tests underlying Section[6](https://arxiv.org/html/2609.36253#S6)\. Files obtained from the original replication package are recorded in a SHA\-256 manifest that is checked before each run\. The manifest, code, configurations, and complete reproduction outputs are provided in the accompanying repository\.

### A\.1Original Design

[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)compare LLM responses with three WVS waves—1995–1997, 2005–2006, and 2017–2018—using four attitudinal items: happiness \(4 ordered options\), political ideology \(10\), religious attendance \(7\), and social trust \(2\)\. Within each country–wave, respondents are stratified by sex, age, education, employment, and marital status\. Rare demographic profiles are merged until each retained group contains at least 20 respondents\. This procedure yields 687 cells for happiness, religious attendance, and social trust and 639 for political ideology, which was not administered in one Russian wave\.

Each cell is represented by an interview\-style prompt containing its demographic profile\. Under next\-token probability \(NTP\) elicitation, the probability mass assigned to valid answer tokens is extracted and renormalized, producing a deterministic response distribution\. Under full\-answer \(FA\) elicitation, answers are sampled at temperature 0\.7 and parsed using a strict response format\. Complete questions and prompt templates appear in Appendix[C\.4](https://arxiv.org/html/2609.36253#A3.SS4)\.

Cell\-level correspondence is measured using the normalized Earth Mover’s Distance in Equation[1](https://arxiv.org/html/2609.36253#S3.E1)\. The measure lies in\[0,1\]\[0,1\]and penalizes disagreement between adjacent response options less than disagreement between distant options\.[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)use thresholds of 0\.05, 0\.10, 0\.15, and 0\.30 to define quality bands from “very good” to “very bad\.” They also compare LLM performance with a leave\-one\-out demographic linear model and random permutations of the survey responses\.

The original study identifies machine bias through three findings\. First, most model–cell predictions fall into the mediocre\-or\-worse nEMD bands, performing worse than the demographic baseline and only slightly better than the random baseline\. Second, multidimensional scaling shows that model cells occupy a narrower response space than survey cells\. For Mixtral\-8x7B on happiness, the median pairwise nEMD is 0\.033 among model cells and 0\.115 among WVS cells\. Third, a cell’s distance from the model’s average response explains much of its prediction error, withR2R^\{2\}values reaching 0\.93–0\.95\. This factor contributes more than the demographic characteristics in 11 of the 12 model–question combinations\.

### A\.2Archived Reproduction

No model inference is performed during the archived reproduction\. We pass the released generations through our independently implemented processing and analysis pipeline and compare the resulting counts, distances, regressions, tables, and figures with the published outputs\. The full\-precision checkpoints available for Mixtral\-8x7B on happiness are reproduced exactly in Table[1](https://arxiv.org/html/2609.36253#A1.T1)\.

Table 1:Our reproduction matches every full\-precision checkpoint reported by[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)for Mixtral\-8x7B on happiness\. Figure[4](https://arxiv.org/html/2609.36253#A1.F4)shows a representative reproduced figure; complete reproduction outputs are provided in the repository\.The pipeline also reproduces Figures 2–4 and 6 from[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6), including the nEMD distributions, social\-coefficient regressions, multidimensional\-scaling geometry, and centroid regressions\. We additionally recover the supplementary analyses of prompting strategy, temperature, and backtranslation for all six published model–elicitation series: GPT\-4T, Llama\-3\-70B, and Mixtral\-8x7B under NTP, and GPT\-3, Llama\-3\-70B, and Mixtral\-8x7B under FA\. Figure[4](https://arxiv.org/html/2609.36253#A1.F4)presents one representative comparison; the complete reproduced outputs are available in the repository\.

Figure 4:Reproduction of Figure 2 from[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6): cell\-level nEMD densities for the six published model–elicitation series, shown with the demographic and random baselines and the original response\-quality bands\.
### A\.3Fresh Reproduction

We regenerate Mixtral\-8x7B responses using the original single\-token, top\-1000\-logprob NTP procedure and FA sampling at temperature 0\.7 with a 12\-token generation budget\. Each prompt produces an atomic record containing its hash, sampling parameters, backend version, and all FA generation attempts\.

The fresh run uses a Q4\_K\_M GGUF quantization of Mixtral\-8x7B\-v0\.1333The exact quantized artifact, repository revision, and checksum are pinned in the accompanying code and verified before each run\.\. Inference usesllama\-cpp\-python0\.3\.1 with CUDA and full GPU offloading\. The run was executed on a single NVIDIA GeForce RTX 5090 \(32 GB\)\.

Archived and fresh runs are compared using difference and equivalence tests with complete country–wave blocks as the resampling unit\. The equivalence margin is±0\.005\\pm 0\.005nEMD, one tenth of the narrowest quality band used in the original study\. Difference\-testpp\-values are obtained from cluster sign\-flip permutations and adjusted within each question across the two elicitation modes using the Holm procedure; equivalence tests are reported without multiplicity adjustment\.

Under NTP, the archived and fresh runs are equivalent within the prespecified margin on all four questions\. For happiness, overall nEMD changes by 2\.9% relative to the archived run\. Under FA, equivalence is inconclusive for happiness and political ideology\. Difference tests detect changes for religious attendance and social trust: mean nEMD changes by\+0\.052\+0\.052and−0\.135\-0\.135, with Holm\-adjustedpp\-values of0\.0100\.010and0\.00020\.0002, respectively\.

We retain both Mixtral series in subsequent analyses\. The archived series preserves comparability with the original study, while the fresh series provides a reproduction reference under the current quantized checkpoint and inference runtime\. Because the fresh run differs from the archived run in both quantization and backend as well as stochastic sampling, archived–fresh differences should not be interpreted as estimates of sampling variability alone\.

### A\.4Additional Reproducibility Controls

For comparability with the original study, each run reproduces its centroid regression on the German cells\. A fixed holdout of 20 cells estimates the model center, avoiding use of the same observations to define the center and fit the regression\. For the remaining cells,log⁡\(1\+nEMD\)\\log\(1\+\\operatorname\{nEMD\}\)error is regressed onlog⁡\(1\+nEMD\)\\log\(1\+\\operatorname\{nEMD\}\)distance from that center\. Holdout indices reproduce the original R random\-number stream using Mersenne–Twister with seed20,240,11020\{,\}240\{,\}110, with stream positions matched by elicitation mode\.

FA generation uses a deterministic seed derived separately for each prompt, so interrupting and resuming a run does not change previously written responses\. A compatibility flag restores the original unseeded sampling configuration when needed\. Unit tests compare the prompt bytes for all four questions and both elicitation modes with those produced by the original implementation\.

## Appendix BPopulation Fidelity Framework and Metric Details

### B\.1Framework Overview

Figure[5](https://arxiv.org/html/2609.36253#A2.F5)summarizes the Population Fidelity framework\. A reference population is partitioned into demographic cells, with each cell represented by an empirical response distributionsis\_\{i\}for a given survey question\. The same demographic profiles are then used to elicit responses from the model, producing corresponding distributionsmim\_\{i\}\.

Population Fidelity compares the survey and model distributions along three dimensions\. Accuracy measures correspondence within individual cells, adaptability compares the amount of variation across cells, and structure measures whether the relative differences among cells are preserved\. Their geometric mean defines the Population Fidelity Score \(PFS\)\. Center alignment compares the equal\-cell survey and model centers and is reported separately\. All quantities can also be recomputed within demographic subsets\.

Reference populationLLMs under evaluationPopulation Fidelitynndemographic cellsfrom survey respondentsdistributionsis\_\{i\}per cellAPIOpen\-weightProprietaryAdapted variantse\.g\. culture, distribution orsubgroup fine\-tuning, etc\.NTP or FA elicitationdistributionmim\_\{i\}per cellAccuracyAdaptabilityStructurePFSgeometric meanCenter alignmentreported separatelyrecomputed for any demographic grouppersonaprompts\{mi\}\\\{m\_\{i\}\\\}\{si\}\\\{s\_\{i\}\\\}123

Figure 5:The Population Fidelity framework compares survey and model distributions cell by cell \(SaccS\_\{\\mathrm\{acc\}\}\), in spread \(SadaptS\_\{\\mathrm\{adapt\}\}\), and in arrangement \(SstructS\_\{\\mathrm\{struct\}\}\); center alignment \(ScenterS\_\{\\mathrm\{center\}\}\) is reported separately\. Section[4](https://arxiv.org/html/2609.36253#S4)defines each\. A reference population providesnndemographic cells with survey distributionssis\_\{i\}\. Model responses to the corresponding demographic profiles produce distributionsmim\_\{i\}\. Accuracy, adaptability, and structure compare the two sets of distributions and combine into the PFS by geometric mean; center alignment is reported separately\. The metrics can also be recomputed within demographic subsets\.
### B\.2Metric Implementation and Validation

This appendix gives the implementation behind every number in Section[6](https://arxiv.org/html/2609.36253#S6): distances, cell retention, degenerate cases, cross\-question aggregation, and the validation checks every run passes\.

##### Distances\.

Equation[1](https://arxiv.org/html/2609.36253#S3.E1)is implemented as the city\-block norm of the difference between the cumulative survey and model distributions, divided byK−1K\-1\. Pairwise distances are stored as condensed vectors using the same cell ordering for survey and model responses\. Although nEMD is normalized to\[0,1\]\[0,1\], differences in response scales and item meaning make averages across questions difficult to interpret\. We therefore compute and interpret all distance\-based quantities separately by question\.

##### Cell\-level model distributions\.

Multiple elicited profiles can map to the same demographic cell\. Under NTP, letℛi\\mathcal\{R\}\_\{i\}denote the profiles assigned to celliiandqi​rq\_\{ir\}the response distribution elicited for profiler∈ℛir\\in\\mathcal\{R\}\_\{i\}, withni​rn\_\{ir\}the number of WVS respondents in celliiwith that profile\. Following[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6), we average over respondents

mi=∑r∈ℛiwi​r​qi​r,wi​r=ni​r∑r′∈ℛini​r′\.m\_\{i\}=\\sum\_\{r\\in\\mathcal\{R\}\_\{i\}\}w\_\{ir\}q\_\{ir\},\\qquad w\_\{ir\}=\\frac\{n\_\{ir\}\}\{\\sum\_\{r^\{\\prime\}\\in\\mathcal\{R\}\_\{i\}\}n\_\{ir^\{\\prime\}\}\}\.Under NTP,qi​rq\_\{ir\}is the renormalized valid\-token probability vector; under FA, one answer is sampled per respondent andmim\_\{i\}is the empirical distribution of valid sampled responses assigned to cellii\.

##### Retention and evaluation support\.

The survey contains 687 demographic cells for happiness, religious attendance, and social trust, and 639 for political ideology\. A cell is scored only when its survey\-response proportions are complete\. Under FA, the cell must also contain at least 20 valid model responses\. Across runs, the retained cell count ranges from 606 to 687\.

Population\-wide analyses use all retained cells\. Target\-country analyses use German cells for German fine\-tuning and Mexican cells for Mexican fine\-tuning\. The complete German subset contains 144 cells and 10,296 cell pairs, while the complete Mexican subset contains 131 cells and 8,515 pairs\. The demographic\-subgroup analysis recomputes all metrics within levels of country, survey wave, sex, age, education, employment, and marital status\. Across all 164 combinations, this yields 4,920 subgroup\-level scores; the main subgroup analysis excludes the two Mixtral reference series, leaving 4,440 scores across 148 combinations\.

For each paired fine\-tuning comparison, the fine\-tuned and as\-released conditions are recomputed on their common set of retained cells\. This prevents differences in response coverage from being interpreted as changes in population fidelity\. The common set is the intersection of the cells retained in both conditions and is used for both population\-level and subgroup\-level comparisons\. Across the primary paired comparisons, these intersections contain between 617 and 687 cells\. Repeated FA runs reuse the same model weights, prompts, decoding configuration, and evaluation procedure, differing only in the seed stream \(Section[5](https://arxiv.org/html/2609.36253#S5)\)\. Retained cell sets are highly stable across repeated runs; Appendix[E\.1](https://arxiv.org/html/2609.36253#A5.SS1)reports the corresponding retention and run\-to\-run variability analyses\.

Under NTP, prompts assigning zero probability mass to all valid answer tokens are treated as failed records rather than renormalized\. No such failure affects the retained cells in the reported analyses\. For conditions evaluated under both NTP and FA, the two elicitation modes use the same retained cell set\.

##### Undefined and degenerate cases\.

Adaptability requires at least two cells and structure at least three\. Below these thresholds, the corresponding quantities remain undefined\. If the model’s pairwise\-distance vector has zero variance,ρ\\rhois undefined; the model is classified as flat and assigned a structure score of zero\. No population\-wide scored run–mode combination meets this condition; two country\-level subgroup estimates do and are assignedSstruct=0S\_\{\\mathrm\{struct\}\}=0under this rule \(Table[8](https://arxiv.org/html/2609.36253#A4.T8)\)\. The nearest case is Qwen3\-VL\-2B\-Thinking under German fine\-tuning on social trust with FA elicitation\. Its median pairwise distance is zero, soA=0A=0, but its distance vector has sufficient variation forρ\\rhoto remain defined\. The PFS equals zero when any component is zero and remains undefined when a required component is undefined\.

##### Cross\-question aggregation\.

The cross\-question PFS is the geometric mean of the four per\-question PFS values\. Component summaries are arithmetic means of their per\-question values, and the limiting component is identified from these averaged scores\. The cross\-question PFS is therefore not recomputed from the averaged components\. Because a zero PFS on any question produces a zero cross\-question geometric mean, per\-question values remain the primary results\.

##### Implementation checks\.

Before producing tables or figures, the implementation verifies that:

1. 1\.every recomputed cell\-level nEMD matches the corresponding run\-level distance table row by row;
2. 2\.pooled quantities match independently calculated population summaries;
3. 3\.DWVSD\_\{\\mathrm\{WVS\}\}is identical across model conditions evaluated on the same question, elicitation mode, and retained cell set;
4. 4\.all transformed component and PFS values lie within\[0,1\]\[0,1\];
5. 5\.group\-level cell counts do not exceed those of their pooled parent populations; and
6. 6\.the survey and model pairwise\-distance vectors contain the same cells in the same order\.

All 164 run–mode combinations pass these checks\.

Center alignment in the Population Fidelity Framework gives equal weight to retained cells\. For comparability with the original study, the reproduction output also preserves its respondent\-weighted center\. Both quantities are retained and labeled according to their weighting scheme\.

### B\.3Stylized Example Calculations

Figure[1](https://arxiv.org/html/2609.36253#S1.F1)uses three subpopulations, A–C\. Their survey distributions are normal with means44,55, and66and standard deviation1\.31\.3, on a 0–10 attitude scale with 101 response options \(0,0\.1,…,100,0\.1,\\ldots,10\)\. The pooled survey distributions¯\\bar\{s\}is their average\.

M1 assignss¯\\bar\{s\}to every group\. M2 re\-sorts the same respondents into three narrower groups, each still one third of respondents, with means3\.343\.34,5\.005\.00, and6\.666\.66and standard deviations0\.810\.81,0\.440\.44, and0\.810\.81\. M3 exchanges the survey distributions of B and C\. Because each model only reassigns the pooled respondents across groups, each averages tos¯\\bar\{s\}, soScenter=1S\_\{\\mathrm\{center\}\}=1for all three models\. Values are rounded to two decimals, derived quantities use unrounded values, and the exact construction is included in the released code\.

The cell\-level errors for M1, M2, and M3 are, respectively,

\[0\.10,0\.02,0\.10\],\[0\.07,0\.07,0\.07\],\[0\.00,0\.10,0\.10\]\.\[0\.10,0\.02,0\.10\],\\qquad\[0\.07,0\.07,0\.07\],\\qquad\[0\.00,0\.10,0\.10\]\.Each therefore has mean errorE≈0\.07E\\approx 0\.07and accuracySacc≈0\.93S\_\{\\mathrm\{acc\}\}\\approx 0\.93\.

Using pair order \(AB, AC, BC\), the pairwise\-distance vectors are

ds=\[0\.10,0\.20,0\.10\],dM1=\[0,0,0\],d\_\{s\}=\[0\.10,0\.20,0\.10\],\\qquad d\_\{\\mathrm\{M1\}\}=\[0,0,0\],dM2=\[0\.17,0\.33,0\.17\],dM3=\[0\.20,0\.10,0\.10\]\.d\_\{\\mathrm\{M2\}\}=\[0\.17,0\.33,0\.17\],\\qquad d\_\{\\mathrm\{M3\}\}=\[0\.20,0\.10,0\.10\]\.Their median pairwise distances are0\.100\.10for the survey,00for M1,0\.170\.17for M2, and0\.100\.10for M3\. The adaptability ratios are thereforeA=0A=0,1\.671\.67, and1\.001\.00, and the adaptability scores are00,0\.600\.60, and1\.001\.00\.

M1’s pairwise distances are all equal, soρ\\rhois undefined and M1 is assignedSstruct=0S\_\{\\mathrm\{struct\}\}=0\(Appendix[B\.2](https://arxiv.org/html/2609.36253#A2.SS2)\)\. Treating distances that are equal by symmetry as tied, the Spearman correlations with the survey distance vector are1\.001\.00for M2 and−0\.50\-0\.50for M3\. After applyingSstruct=max⁡\(0,ρ\)S\_\{\\mathrm\{struct\}\}=\\max\(0,\\rho\), the structure scores are00,1\.001\.00, and00\. Finally,

PFSM1=0,PFSM2≈0\.82,PFSM3=0\.\\mathrm\{PFS\}\_\{\\mathrm\{M1\}\}=0,\\qquad\\mathrm\{PFS\}\_\{\\mathrm\{M2\}\}\\approx 0\.82,\\qquad\\mathrm\{PFS\}\_\{\\mathrm\{M3\}\}=0\.Thus, the three models reproduce the pooled survey distribution exactly and have nearly identical accuracy, but differ in the amount and structure of their between\-group variation\. M1 corresponds to the survey\-center baseline \(Appendix[D\.1](https://arxiv.org/html/2609.36253#A4.SS1)\)\.

## Appendix CExperimental Configuration and Prompts

### C\.1Models and Fine\-Tuning Configuration

##### Model inventory\.

Table[2](https://arxiv.org/html/2609.36253#A3.T2)summarizes the six open\-weight models used in the main evaluation\. Each is evaluated as released and after German and Mexican fine\-tuning\. Within each model, the as\-released and fine\-tuned conditions use the same tokenizer, numerical configuration, and inference settings\. Exact repository revisions are recorded in the released run manifests\.

Mixtral\-8x7B is retained separately as a reproduction reference\. The archived condition uses generations released with the original Machine Bias replication package, while the fresh condition regenerates responses from Mixtral\-8x7B\-v0\.1 using the original prompts and elicitation procedures\. All local open\-weight training and inference experiments were run on a single NVIDIA GeForce RTX 5090 \(32 GB\)\. The backend used for the fresh Mixtral reproduction is described in Appendix[A](https://arxiv.org/html/2609.36253#A1)\.

Table 2:Open\-weight model inventory\. NF4 is 4\-bit NormalFloat quantization implemented withbitsandbytes0\.49\.2\. Adapted\-modules column reports the number of modules receiving LoRA updates\. Every model runs in bfloat16 compute withPEFT0\.19\.1 and PyTorch 2\.12\.0 under Python 3\.11\.
##### Fine\-tuning configuration\.

For each target culture, the training data combine the WVS seed questions selected by CultureLLM with semantically augmented paraphrases\. Each record is paired with a culture\-specific target answer derived from aggregate survey responses\. The German condition uses CultureLLM’s German partition\. For the Mexican condition, we reconstruct the records using only Mexican respondents rather than CultureLLM’s original Spanish partition, which pools respondents from Mexico and Argentina\.

Fine\-tuning uses LoRA with rankr=8r=8, scaling parameterα=16\\alpha=16, and dropout0\.050\.05\. Adapters are applied to the query, key, value, and output projections of the text\-attention blocks \(q\_proj,k\_proj,v\_proj, ando\_proj\); vision and audio components are excluded\. QLoRA with NF4 quantization is used for Gemma\-4\-31B\-it and Muse\-Glimmer\-30B\. Within each model family, the German and Mexican variants use the same training configuration\. None of the four evaluation questions appears in the fine\-tuning data\.

Training uses supervised fine\-tuning on chat\-formatted records with AdamW, no weight decay, a cosine learning\-rate schedule, an effective batch size of 16, gradient clipping at 0\.3, a maximum sequence length of 512 tokens, and seed 42\. Warmup lasts 50 optimizer steps, or 100 steps for the QLoRA models\. Training is allowed to continue for at most 250 epochs and is stopped early based on validation loss after a minimum of 20 epochs, with patience of 15 evaluations\. Validation is performed every 100 optimizer steps, and the checkpoint with the lowest validation loss is retained\. Table[3](https://arxiv.org/html/2609.36253#A3.T3)reports the learning rate, batch size, run length and held\-out accuracy of each model\.

Table 3:Fine\-tuning configuration and run length by model and target culture\. Batch reports the per\-device batch size×\\timesgradient\-accumulation steps, yielding an effective batch size of 16\. Parentheses report optimizer steps at termination\. Val\. acc\. is the held\-out token accuracy of the retained checkpoint\.The German dataset contains 2,100 question–answer pairs, split into 1,890 training and 210 validation records\. The Mexican dataset is constructed from the same 50 WVS seed questions and 1,000 augmented paraphrases, using aggregate responses from Mexican respondents only\. It contains 1,050 pairs, split into 945 training and 105 validation records\.

All fine\-tuning records are in English: each contains a system prompt naming the target country, a WVS\-derived question with numbered response options, and the corresponding target answer\. The experiment therefore evaluates cultural adaptation within a common language rather than cross\-lingual transfer\. Training was performed on the hardware described in Appendix[A](https://arxiv.org/html/2609.36253#A1)\.

### C\.2Response Classification, Capacity, and Coverage

##### Response classification\.

Each FA generation is classified as*valid*if it maps unambiguously to one offered response option,*invalid*if it does not satisfy the required format, and*failed*if no generation is completed\. Invalid and failed outputs are retained for audit\. For NTP, valid\-token probability mass is measured before renormalization\. A response assigning no probability mass to any valid token is recorded as degenerate rather than silently renormalized\. Coverage records and the original generated outputs are included in the repository\.

##### Capacity and response coverage\.

Each run receives a 32\-prompt preflight probe\. Under NTP, a run is flagged as non\-informative if the mean probability mass assigned to valid answer tokens is below 0\.10\. No NTP run in the reported evaluation falls below this threshold\.

Under FA, we do not apply a separate run\-level threshold\. Distances are computed only for cells containing at least 20 valid responses, and dropped cells are recorded for each run\. Across the evaluated FA conditions, the lowest overall valid\-answer rate is 0\.926\.

##### Fine\-tuning health\.

Before elicitation, every adapter used in the reported experiments is checked using the mean norm of the weight update applied to the attention projections and held\-out token accuracy against a 50% usability threshold\. All evaluated adapters pass this check, with update norms between 0\.2 and 5\.4 and held\-out accuracy between 77% and 98%\. No reported Population Fidelity result therefore comes from a checkpoint classified as a training failure\.

### C\.3API\-Served Model

##### API\-served model\.

GPT\-5\.6 Terra is evaluated through a commercial chat endpoint using the same survey prompts as the open\-weight models\. Requests use the model identifiergpt\-5\.6\-terraand were issued in August 2026\. Reasoning effort is fixed at*medium*, FA responses are sampled at temperature 0\.7, and the maximum output length is 300 tokens\. A seed of 20240110 is passed with each request as a best\-effort parameter\.

At nonzero reasoning effort, the endpoint does not provide usable next\-token probabilities for our NTP procedure\. In a single\-token probe requesting the top 20 log probabilities, none of the valid survey\-answer tokens appeared among the returned candidates\. Terra is therefore evaluated under FA only\.

Each request records the prompt, requested sampling parameters, parameters applied by the endpoint, and returned output\. Prompts are submitted as a single user message rather than as raw completions\.

##### Computational scale\.

The experiment requires approximately7\.97\.9million model queries, excluding the archived Mixtral generations\. Each open\-weight NTP run evaluates 13,904 unique demographic profiles, while each FA run evaluates 26,981 respondent\-level prompts\.

### C\.4Questions and Prompt Construction

Every response analysed in this paper comes from the four survey items and the prompt template below, used as described in Sections[3](https://arxiv.org/html/2609.36253#S3)and[5\.3](https://arxiv.org/html/2609.36253#S5.SS3)\.

Prompts for the four items in Table[4](https://arxiv.org/html/2609.36253#A3.T4)follow the interview\-style template used by[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)\. As illustrated in Figure[6](https://arxiv.org/html/2609.36253#A3.F6), the interviewer asks a sequence of sociodemographic questions and the interviewee supplies the values defining the relevant cell\. The model then completes the answer to the final attitudinal question\. Categorical demographic questions use lettered options and expect a single capital letter\. Under NTP elicitation, the probability mass assigned to the valid first\-token options is extracted and renormalized\.

Table 4:Attitudinal items used in the evaluation\.Question: What year is it? Answer: \{\{survey year\}\} Question: What country are we in? Is it ’A\. Australia’ or ’B\. Germany’ or ’C\. Mexico’ or ’D\. Russia’ or ’E\. United States’? Answer: \{\{lettered country\}\} Question: How old are you? Answer: \{\{age\}\} Question: What is your gender? Is it ’A\. Male’ or ’B\. Female’? Answer: \{\{lettered sex\}\} Question: What is the highest educational level that you have attained? Is it ’A\. Low’ or ’B\. Middle’ or ’C\. High’? Answer: \{\{lettered education\}\} Question: Are you employed now or not? Are you ’A\. Working’ or ’B\. Retired’ or ’C\. Homemaker’ or ’D\. Student’ or ’E\. Unemployed’? Answer: \{\{lettered employment\}\} Question: What is your marital status? Are you ’A\. Married’ or ’B\. Cohabiting’ or ’C\. Divorced or separated’ or ’D\. Widowed’ or ’E\. Single’? Answer: \{\{lettered marital status\}\} Question: \{\{attitudinal item\}\} Answer:*The attitudinal item, per question:*d\_happy: Taking all things together, how happy are you? Are you ’A\. Very happy’ or ’B\. Quite happy’ or ’C\. Not very happy’ or ’D\. Not at all happy’?d\_polpos: In political matters, people talk of "the left" and "the right\." How would you place your views on a scale of \{\{low\}\} to \{\{high\}\}, where \{\{low\}\} is left and \{\{high\}\} is right, generally speaking?\(\{\{low\}\}–\{\{high\}\} is 0–9 for NTP and 1–10 for FA\)d\_religiousp: Apart from weddings, funerals and christenings, about how often do you attend religious services these days? Is it ’A\. More than once a week’ or ’B\. Once a week’ or ’C\. Once a month’ or ’D\. Only on special holy days’ or ’E\. Once a year’ or ’F\. Less often’ or ’G\. Never, practically never’?d\_trust: Generally speaking, would you say that most people can be trusted or that you need to be very careful in dealing with people? Is it ’A\. Most people can be trusted’ or ’B\. Need to be very careful’?

Figure 6:Interview\-style prompt template\. Placeholders are replaced with the demographic values defining each cell; survey year and age are inserted as integers, while categorical attributes use the corresponding lettered options\. Demographic items absent from a merged profile are omitted\. The final question is replaced by one of the four attitudinal items in Table[4](https://arxiv.org/html/2609.36253#A3.T4), and the prompt ends atAnswer:for the model to complete\.Political ideology follows the original study’s mode\-specific encoding\. NTP uses a 0–9 scale so that each option corresponds to a single digit token, whereas FA uses the survey’s 1–10 scale\. The two encodings are aligned by ordinal position before the response distributions are compared\. The NTP prompt for political ideology includes a trailing space afterAnswer:to preserve the original tokenization\. Political ideology is unavailable for 48 cells from a survey wave in which the item was not administered; these cells are treated as having undefined targets and excluded from analyses of that question\. Unit tests verify the prompt bytes for all four questions and both elicitation modes against the original implementation\.

## Appendix DSupplementary Results

### D\.1Survey\-Center Baseline for Accuracy

Accuracy is bounded, and on the nEMD scale a prediction that ignores demographic information can already attain a high score\. We therefore compare model accuracy with such a baseline\. For the retained cellsIIof a run, we take the equal\-cell survey centers¯I\\bar\{s\}\_\{I\}defined in Section[4\.3](https://arxiv.org/html/2609.36253#S4.SS3)and predict it for every cell,mi\(0\)=s¯Im\_\{i\}^\{\(0\)\}=\\bar\{s\}\_\{I\}for alli∈Ii\\in I\. Its accuracy is

Sacc\(0\)=1−1\|I\|​∑i∈InEMD⁡\(si,s¯I\),S\_\{\\mathrm\{acc\}\}^\{\(0\)\}=1\-\\frac\{1\}\{\|I\|\}\\sum\_\{i\\in I\}\\operatorname\{nEMD\}\\left\(s\_\{i\},\\bar\{s\}\_\{I\}\\right\),which is the accuracy attained without using demographic information\. We also express model performance relative to this baseline using the skill\-score form

Saccabove​baseline=Sacc−Sacc\(0\)1−Sacc\(0\)\.S\_\{\\mathrm\{acc\}\}^\{\\mathrm\{above\\,baseline\}\}=\\frac\{S\_\{\\mathrm\{acc\}\}\-S\_\{\\mathrm\{acc\}\}^\{\(0\)\}\}\{1\-S\_\{\\mathrm\{acc\}\}^\{\(0\)\}\}\.This score is zero at the survey\-center baseline, one for perfect cell\-level prediction, positive when a model exceeds the baseline, and negative when it performs worse\.

We score the baseline using the same implementation as the model conditions, so its other components follow directly from the definitions in Section[4](https://arxiv.org/html/2609.36253#S4)\. Its center alignment is one by construction\. Because all predicted cells are identical, all pairwise model distances are zero, givingA=0A=0andSadapt=0S\_\{\\mathrm\{adapt\}\}=0; the flatness rule of Appendix[B\.2](https://arxiv.org/html/2609.36253#A2.SS2)setsSstruct=0S\_\{\\mathrm\{struct\}\}=0, and thereforePFS=0\\mathrm\{PFS\}=0\. Becauses¯I\\bar\{s\}\_\{I\}includes each cell’s own survey distribution, we also compute a leave\-one\-out version in which celliiis excluded from the center used to predict it\. This changesSacc\(0\)S\_\{\\mathrm\{acc\}\}^\{\(0\)\}by less than0\.00030\.0003at the population level and by at most0\.0050\.005within demographic levels, indicating that the baseline’s performance is not driven by including each cell in its own reference center\.

Table[5](https://arxiv.org/html/2609.36253#A4.T5)compares the baseline with the models, with each run evaluated on its own retained cells\. The survey\-center baseline attainsSacc\(0\)S\_\{\\mathrm\{acc\}\}^\{\(0\)\}between0\.8480\.848for religious attendance and0\.9150\.915for political ideology and exceeds the median model accuracy for every question and elicitation mode\. Only 7 of the 164 combinations exceed their corresponding baseline, by at most0\.0360\.036in accuracy: on happiness, both Mixtral runs under NTP, the regenerated Mixtral run under FA, and the as\-released and Mexican Muse\-Glimmer\-30B conditions under NTP; on religious attendance, the Mexican Muse\-Glimmer\-30B condition under NTP; and on social trust, as\-released Qwen3\-VL\-8B under NTP\. The median above\-baseline score is−0\.56\-0\.56, and 52 combinations fall below−1\-1, meaning that their mean cell\-level error is more than twice that of the corresponding survey\-center baseline\.

QuestionModeCellsSacc\(0\)S\_\{\\mathrm\{acc\}\}^\{\(0\)\}ModelSaccS\_\{\\mathrm\{acc\}\}Saccabove​baselineS\_\{\\mathrm\{acc\}\}^\{\\mathrm\{above\\,baseline\}\}Above baselineHappinessFA678–6870\.901–0\.9020\.860 \[0\.749, 0\.906\]−0\.41\-0\.41\[−1\.53\-1\.53, 0\.05\]1 / 21NTP678–6870\.901–0\.9020\.871 \[0\.769, 0\.911\]−0\.30\-0\.30\[−1\.33\-1\.33, 0\.11\]4 / 20Political ideologyFA606–6390\.913–0\.9150\.875 \[0\.817, 0\.905\]−0\.45\-0\.45\[−1\.11\-1\.11,−0\.09\-0\.09\]0 / 21NTP626–6390\.913–0\.9140\.892 \[0\.848, 0\.910\]−0\.25\-0\.25\[−0\.77\-0\.77,−0\.03\-0\.03\]0 / 20Religious attendanceFA677–6870\.8480\.707 \[0\.441, 0\.833\]−0\.92\-0\.92\[−2\.67\-2\.67,−0\.10\-0\.10\]0 / 21NTP677–6870\.8480\.754 \[0\.480, 0\.884\]−0\.61\-0\.61\[−2\.41\-2\.41, 0\.24\]1 / 20Social trustFA630–6870\.867–0\.8680\.683 \[0\.350, 0\.855\]−1\.39\-1\.39\[−3\.90\-3\.90,−0\.09\-0\.09\]0 / 21NTP630–6870\.867–0\.8680\.703 \[0\.359, 0\.883\]−1\.24\-1\.24\[−3\.84\-3\.84, 0\.11\]1 / 20All questionsboth606–6870\.848–0\.9150\.832 \[0\.350, 0\.911\]−0\.56\-0\.56\[−3\.90\-3\.90, 0\.24\]7 / 164Demographic levelsboth24–3770\.817–0\.9430\.824 \[0\.185, 0\.952\]−0\.72\-0\.72\[−11\.58\-11\.58, 0\.40\]157 / 4,440Target countriesboth130–1440\.875–0\.9430\.816 \[0\.384, 0\.942\]−1\.24\-1\.24\[−4\.89\-4\.89, 0\.01\]1 / 96

Table 5:Model accuracy relative to the survey\-center baseline\.Sacc\(0\)S\_\{\\mathrm\{acc\}\}^\{\(0\)\}is obtained by predicting the equal\-cell survey center for every retained cell, so its range reflects differences in retained cells across runs\. ModelSaccS\_\{\\mathrm\{acc\}\}and the above\-baseline score report the median across combinations, with the minimum and maximum in brackets; the final column counts combinations whose accuracy exceeds their corresponding baseline\. For demographic\-level results, the baseline is recomputed within each level\. Target\-country results compare German fine\-tuned conditions within Germany and Mexican fine\-tuned conditions within Mexico\. The baseline hasPFS=0\\mathrm\{PFS\}=0in every case\.The same pattern holds within demographic levels, where each level is compared with the center of its own retained cells\. Across the 4,440 subgroup\-level scores of Section[6](https://arxiv.org/html/2609.36253#S6), models exceed their level\-specific survey\-center baseline in 157 cases\. Within the target countries, German fine\-tuned conditions never exceed the German survey\-center baseline \(0 of 48 comparisons\), while Mexican fine\-tuned conditions exceed the Mexican baseline once \(1 of 48\)\.

This baseline clarifies why high accuracy does not imply population fidelity\. Although models exceed the baseline’s structure score of zero in 134 of the 164 combinations, they exceed its accuracy in only seven\. Much of their absolute accuracy therefore reflects proximity to the survey center, consistent with the machine\-bias pattern of[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)\. The baseline also illustrates the behavior that PFS is designed to reject: a prediction that outperforms nearly all evaluated combinations on accuracy and matches the survey center exactly receivesPFS=0\\mathrm\{PFS\}=0because it contains no variation between groups\.

We report the above\-baseline score as a diagnostic and do not include it in PFS\. Within each question, the baseline varies by at most0\.00150\.0015across runs, so the above\-baseline score produces almost the same ordering asSaccS\_\{\\mathrm\{acc\}\}, with Spearman correlations of at least0\.9980\.998\. It therefore changes the reference point while barely affecting model rankings\. Unlike the PFS components, which lie in\[0,1\]\[0,1\], the above\-baseline score can take large negative values when model error substantially exceeds baseline error; in our population\-level results it reaches−3\.90\-3\.90\.

### D\.2Model\- and Question\-Specific Results

Under FA elicitation, fine\-tuned conditions attain the highest PFS on three of the four questions, all with Muse\-Glimmer\-30B: its German condition leads on political ideology and social trust, while its Mexican condition ties GPT\-5\.6 Terra on religious attendance after rounding\. Happiness is led by as\-released Qwen3\-VL\-2B \(Table[6](https://arxiv.org/html/2609.36253#A4.T6)\)\.

Table 6:Population Fidelity Score by question under FA elicitation\. The highest score in each column is shown in bold\. GPT\-5\.6 Terra and Muse\-Glimmer\-30B \(ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}\) both score 0\.726 on religious attendance after rounding\. GPT\-3 \(archived\) denotes the proprietary FA series from the original Machine Bias study, evaluated with the same Population Fidelity metrics on its archived outputs\.Averaged across questions and elicitation modes, only Gemma\-4\-E4B and Muse\-Glimmer\-30B improve under both cultural conditions\. Gemma\-4\-E4B has meanΔ​PFS\\Delta\\mathrm\{PFS\}of\+0\.063\+0\.063underℱde\\mathcal\{F\}\_\{\\text\{de\}\}and\+0\.050\+0\.050underℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}, while Muse\-Glimmer\-30B improves by\+0\.046\+0\.046and\+0\.025\+0\.025, respectively\. Qwen3\-VL\-2B declines by−0\.114\-0\.114and−0\.109\-0\.109and improves in none of its eightℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}comparisons\. Fine\-tuning effects also vary substantially by question:ℱde\\mathcal\{F\}\_\{\\text\{de\}\}improves eight of 12 comparisons for happiness but three for social trust, whileℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}improves two for happiness and seven for political ideology\. No model improves on all four questions under both modes for either target\.

Those PFS changes hide components that move in different directions\. Mean changes over the 48 comparisons, on all demographic cells and forℱde\\mathcal\{F\}\_\{\\text\{de\}\}thenℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}, are−0\.077\-0\.077and\+0\.014\+0\.014forSaccS\_\{\\mathrm\{acc\}\},−0\.103\-0\.103and−0\.064\-0\.064forSadaptS\_\{\\mathrm\{adapt\}\}, and\+0\.008\+0\.008and\+0\.004\+0\.004forSstructS\_\{\\mathrm\{struct\}\}; Section[6\.3](https://arxiv.org/html/2609.36253#S6.SS3)gives them within the target countries\.

##### Comparison with archived proprietary\-model outputs\.

The archived outputs from the original Machine Bias study provide a descriptive reference for comparing GPT\-5\.6 Terra with an earlier proprietary model under FA elicitation\.[Boelaert et al\. \(2025\)](https://arxiv.org/html/2609.36253#bib.bib6)usedavinci\-002, referred to as GPT\-3 in the original study, for their FA robustness analysis\. We evaluate those archived outputs with the same Population Fidelity metrics used for Terra\. This comparison is observational rather than controlled: the models differ in model vintage, post\-training, interface, and generation procedure, so observed differences cannot be attributed to any single factor; the corresponding scores appear in Table[6](https://arxiv.org/html/2609.36253#A4.T6)\.

Terra attains a higher PFS on three of the four questions: happiness \(0\.3730\.373vs\.0\.2450\.245\), religious attendance \(0\.7260\.726vs\.0\.0000\.000\), and social trust \(0\.4160\.416vs\.0\.2210\.221\)\. Its PFS is lower for political ideology \(0\.2700\.270vs\.0\.3170\.317\)\. The largest difference is in structure, where Terra scores higher on three questions and is nearly unchanged on political ideology\.

Figure[7](https://arxiv.org/html/2609.36253#A4.F7)shows the same comparison within demographic levels\. Across 120 level\-specific comparisons, Terra has higher structure in 109 and higher PFS in 86, but higher center alignment in only 54 and higher adaptability in 25\. Thus, in this descriptive comparison, Terra more often recovers the observed organization of differences among groups, without consistently improving either the amount of between\-group variation or agreement with the survey center\.

Figure 7:PFS and center alignment by demographic level for the two proprietary models under FA elicitation, shown separately by question\. Within each question, the upper row shows PFS and the lower row center alignment; demographic columns follow Figure[3](https://arxiv.org/html/2609.36253#S6.F3)\. Blue circles denote archived GPT\-3 outputs from the original study, and green triangles denote GPT\-5\.6 Terra as released\.

### D\.3Accuracy and Adaptability by Question

Compression dominates, with 43 of the 52 as\-released run–mode combinations atA<1A<1and only 9 atA\>1A\>1\(Table[7](https://arxiv.org/html/2609.36253#A4.T7)\)\. All 26 happiness and political ideology combinations compress; amplification occurs only for religious attendance and social trust\. Only one of the nine amplifying combinations reachesρ\>0\.1\\rho\>0\.1, illustrating why adaptability and structure must be examined separately\.

Table 7:Compression is universal for happiness and political ideology, while amplification occurs only for religious attendance and social trust, as theA<1A<1andA\>1A\>1columns show by counting conditions producing less and more between\-group variation than the survey\. Counts cover the six open\-weight LLMs under both elicitation modes and GPT\-5\.6 Terra under FA, excluding the Mixtral reference series\.Figure[8](https://arxiv.org/html/2609.36253#A4.F8)complements these counts by showing howSaccS\_\{\\mathrm\{acc\}\}relates toAAunder FA elicitation\. Conditions nearA=1A=1reproduce the survey’s overall amount of between\-group variation, but this does not establish that the variation is assigned to the corresponding social groups\.

Figure 8:Cell\-level error and adaptability under FA elicitation\. The horizontal axis reports mean nEMD to the WVS cells, with lower values indicating greater accuracy\. The vertical axis reports the adaptability ratioAA\. Values belowA=1A=1indicate compression of between\-group variation, while values above one indicate amplification\. Adaptability must be interpreted alongside structure because matching the amount of variation does not establish that it occurs between the corresponding social groups\.
### D\.4Demographic Breakdown

The demographic\-subgroup analysis contains 1,440 paired scores for each cultural target\. German fine\-tuning improves PFS in 42\.5% of these comparisons, reduces it in 49\.7%, and leaves 7\.8% unchanged\. The corresponding proportions for Mexican fine\-tuning are 40\.8%, 48\.5%, and 10\.7%\.

Improvements are not concentrated in the target countries\. Within Germany, German fine\-tuning improves 18 of 48 comparisons, with a mean PFS change of−0\.021\-0\.021\. Across the other countries, it improves 38\.0% of comparisons, with a mean change of−0\.019\-0\.019\. Within Mexico, Mexican fine\-tuning also improves 18 of 48 comparisons, with a mean change of−0\.010\-0\.010\. Outside Mexico, it improves 32\.3%, with a mean change of−0\.035\-0\.035\.

![Refer to caption](https://arxiv.org/html/2609.36253v1/fig_fidelity_country_shift_happiness_fa.png)Figure 9:Change in PFS by country after cultural fine\-tuning for happiness under FA elicitation\. Each row represents a fine\-tuned condition and each column a surveyed country\. Green indicates improvement relative to the corresponding as\-released model, and red indicates decline\. For happiness under FA, target\-country changes are not consistently more favorable than changes in the other surveyed countries\.For each condition and demographic family, we define the within\-condition spread as the maximum minus minimum PFS across that family’s levels and average this quantity across conditions\. The mean within\-condition spread in PFS is largest across age \(0\.2870\.287\) and employment \(0\.2700\.270\), followed by country \(0\.2540\.254\), marital status \(0\.2410\.241\), education \(0\.1790\.179\), survey wave \(0\.1070\.107\), and sex \(0\.0540\.054\)\. Although adaptability varies substantially across levels, PFS spread is more sensitive to structure: holding each condition’s structure at its family mean reduces the mean spread to between0\.0180\.018and0\.0800\.080, whereas holding adaptability at its family mean leaves it between0\.0480\.048and0\.2690\.269\.

Mean PFS generally increases with age, although not monotonically\. It is0\.2170\.217and0\.2070\.207in the two youngest groups and0\.3100\.310and0\.3170\.317in the two oldest\. Students have the lowest mean PFS among employment groups \(0\.1710\.171\), compared with0\.2740\.274for retired respondents\. These estimates should be interpreted cautiously because the oldest and student groups contain comparatively few cells and cell pairs\. Table[8](https://arxiv.org/html/2609.36253#A4.T8)reports the complete results and sample sizes\.

FamilyLevelPFSSaccS\_\{\\mathrm\{acc\}\}SadaptS\_\{\\mathrm\{adapt\}\}SstructS\_\{\\mathrm\{struct\}\}CellsPairsCountryUnited States0\.2900\.7830\.5080\.0901409,674Germany0\.2730\.7730\.5270\.09314410,243Australia0\.2700\.7720\.5490\.0791369,126Mexico0\.2240\.7490\.5110\.0631318,461Russia0\.2190\.7550\.4880\.0601247,817Age<<250\.2170\.7770\.4470\.079772,94125–340\.2070\.7620\.4670\.07215712,23835–440\.2350\.7620\.4840\.1011116,14945–540\.2640\.7630\.5070\.1111146,50555–640\.2630\.7610\.5050\.093974,65565–740\.3100\.7800\.5100\.117873,71375\+0\.3170\.7860\.5220\.14528373SexFemale0\.2480\.7710\.4970\.09336767,199Male0\.2370\.7620\.4980\.08330646,805EducationLow0\.2710\.7600\.4970\.109511,268High0\.2150\.7690\.4960\.07626133,965Middle0\.2050\.7660\.4980\.06636165,196Employ\.Homemaker0\.2750\.7680\.4530\.120742,718Retired0\.2740\.7790\.5100\.09315411,819Working0\.2410\.7580\.4700\.08836968,014Unemployed0\.2320\.7780\.4760\.09133527Student0\.1710\.7940\.4360\.06538699Marital st\.Cohabiting0\.2730\.7660\.3750\.15238721Married0\.2650\.7540\.4480\.10036064,753Widowed0\.2510\.7870\.4140\.100652,108Div\. / sep\.0\.2330\.7950\.4440\.079732,631Single0\.2020\.7760\.4250\.0711369,220Wave1995–19970\.2460\.7620\.5070\.08622124,2182005–20060\.2500\.7670\.4950\.10020821,8392017–20180\.2490\.7710\.5190\.08824429,700

Table 8:Demographic\-subgroup Population Fidelity\. Metrics are recomputed within each level and then averaged across the 148 run–mode combinations, excluding the archived and fresh Mixtral reference series\. Cells and pairs report the mean numbers contributing to each estimate\. Estimates based on fewer cells, especially structure, should be interpreted cautiously\. Two country\-level estimates have zero\-variance model distance vectors and receive a structure score of zero under the flatness rule in Appendix[B\.2](https://arxiv.org/html/2609.36253#A2.SS2)\.##### Center alignment and PFS by demographic level\.

Figure[3](https://arxiv.org/html/2609.36253#S6.F3)compares PFS and center alignment across demographic levels for happiness, while Figure[10](https://arxiv.org/html/2609.36253#A4.F10)extends the comparison to all four questions\. Center alignment varies much less across demographic levels than PFS\. Excluding the Mixtral reference series, the median center alignment across the 30 levels ranges from0\.8350\.835for cohabiting respondents to0\.8780\.878for Australia, whereas median PFS ranges from0\.1430\.143for students to0\.3430\.343for respondents aged 75 and older\. For happiness alone, the corresponding ranges are0\.8480\.848–0\.9280\.928for center alignment and0\.1780\.178–0\.4360\.436for PFS\.

This difference is also visible within particular demographic levels\. Students, unemployed respondents, respondents aged 25–34, and Russian cells have median center alignment within0\.010\.01of the pooled value \(0\.8600\.860\), but substantially lower median PFS \(0\.1430\.143–0\.2380\.238, compared with0\.2540\.254for the pooled population\)\. Among the 4,440 subgroup\-level scores excluding Mixtral references, 295 combine center alignment of at least0\.850\.85with PFS of at most0\.100\.10\. Structure is the limiting component in 278 of these cases\. A model can therefore approximate a subgroup center while recovering little of the pattern of differences among the cells that compose it\.

Across model conditions, PFS and center alignment are positively but only moderately associated\. Excluding Mixtral references, the mean Spearman correlation across demographic levels ranges from0\.2630\.263for political ideology under NTP to0\.4250\.425for social trust under NTP\. At the pooled population level, correlations range from0\.3020\.302for religious attendance under FA to0\.6740\.674for political ideology under FA\. Bootstrap intervals are wide, particularly at the population level\. Pearson correlations give the same qualitative pattern\. These results support treating center alignment and Population Fidelity as related but distinct quantities\.

Figure 10:PFS and center alignment by demographic level under FA elicitation, shown separately by question\. Filled boxes show the distribution of PFS and hollow boxes the distribution of center alignment across 19 model conditions; Mixtral reference series are excluded\. Demographic columns follow Figure[3](https://arxiv.org/html/2609.36253#S6.F3)\.Figure 11:PFS and center alignment by demographic level for individual model conditions under FA elicitation, shown separately by question\. Within each question, the upper row shows PFS and the lower row center alignment; demographic columns follow Figure[3](https://arxiv.org/html/2609.36253#S6.F3)\. Color and marker shape identify the model\. Hollow markers denote as\-released conditions, filled markers German fine\-tuning, and half\-filled markers Mexican fine\-tuning\.The converse pattern also occurs, although less often\. A PFS of at least0\.400\.40together with center alignment of at most0\.750\.75occurs 84 times, 60 of them for social trust\. Fine\-tuned conditions account for 60 of these cases\. Thus, relatively high subgroup fidelity can also coexist with weaker center alignment\. Figure[11](https://arxiv.org/html/2609.36253#A4.F11)shows that these differences are distributed across model conditions rather than being driven by a single model\.

### D\.5Additional Population Fidelity Figures

Figure[12](https://arxiv.org/html/2609.36253#A4.F12)relates the amount of model variation to its correspondence with observed group differences\. Figures[14](https://arxiv.org/html/2609.36253#A4.F14)and[15](https://arxiv.org/html/2609.36253#A4.F15)extend the main\-text fine\-tuning and component figures to the remaining FA questions and the cross\-question summary\. Complete NTP and FA figures are available in the accompanying repository\.

Figure 12:Adaptability and structure under NTP elicitation, shown separately by question\. The horizontal axis reports the adaptability ratioAA, and the vertical axis the untransformed structure correlationρ\\rho\. Dashed lines markA=1A=1andρ=0\\rho=0\. Dotted connectors link each population\-wide estimate to the corresponding estimate recomputed within the target\-country subset: Germany for German fine\-tuning and Mexico for Mexican fine\-tuning\. As\-released conditions show both target\-country estimates, whereas fine\-tuned conditions show only the estimate for their target country\. Conditions withA\>1A\>1andρ≈0\\rho\\approx 0exhibit amplified between\-group variation without recovering the observed structure of group differences\.\(a\)Component profile, all retained cells\.\(b\)Fine\-tuning shifts, all retained cells\.
Figure 13:Population Fidelity for happiness under FA elicitation on all retained cells\.\(a\)Component scores for all 21 conditions, ordered by PFS; diamonds show PFS and hollow squares center alignment\.\(b\)Fine\-tuned minus as\-released changes computed on the cells retained by both conditions\. Figure[2](https://arxiv.org/html/2609.36253#S6.F2)shows the corresponding target\-country results\. Plotting conventions are as in Figure[2](https://arxiv.org/html/2609.36253#S6.F2)\.Figure 14:Changes in PFS and center alignment after cultural fine\-tuning under FA elicitation, for the cross\-question summary, political ideology, religious attendance, and social trust\. Each point compares a fine\-tuned condition with its corresponding as\-released model\. Movement to the right indicates improved PFS, and upward movement indicates improved center alignment\. The happiness result appears in Figure[2\(b\)](https://arxiv.org/html/2609.36253#S6.F2.sf2)\.\(a\)Cross\-question summary\.\(b\)Political ideology\.\(c\)Religious attendance\.\(d\)Social trust\.
Figure 15:Population Fidelity components under FA elicitation, for the cross\-question summary, political ideology, religious attendance, and social trust\. Accuracy is generally the strongest component, whereas structure is the weakest and most often limiting; happiness is shown in Figure[2\(a\)](https://arxiv.org/html/2609.36253#S6.F2.sf1)and movement intervals in Table[9](https://arxiv.org/html/2609.36253#A5.T9)\. Fine\-tuned conditions are labelled German \(ℱde\\mathcal\{F\}\_\{\\text\{de\}\}\) and Mexican \(ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}\)\.

## Appendix ERobustness and Sensitivity

### E\.1Stability Across Repeated Full\-Answer Runs

Because FA elicitation is stochastic, repeated runs can produce different sampled responses and therefore slightly different scores\. We quantify this run\-to\-run variation for the 18 locally evaluated conditions—the as\-released,ℱde\\mathcal\{F\}\_\{\\text\{de\}\}, andℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}variants of six models—on all four questions, yielding 72 condition–question pairs evaluated three times each\. Across repeats, model weights, fine\-tuned variants, prompts, decoding settings, and library versions are held fixed; only the random seed stream changes\. NTP is deterministic under our decoding configuration and is therefore not repeated\.

For each condition–question pairii, we compute the mean score across itsc=3c=3FA runs and a 95%tt\-interval,

x¯i±hi,hi=t0\.975,c−1​σic,\\bar\{x\}\_\{i\}\\pm h\_\{i\},\\qquad h\_\{i\}=t\_\{0\.975,\\,c\-1\}\\frac\{\\sigma\_\{i\}\}\{\\sqrt\{c\}\},whereσi\\sigma\_\{i\}is the standard deviation across the three runs andhih\_\{i\}is the corresponding CI half\-width\. Becausec=3c=3,t0\.975,2=4\.30t\_\{0\.975,2\}=4\.30, so individual intervals can be wide even when the observed run\-to\-run variation is modest\.

To summarize run\-to\-run uncertainty across a set ofnncondition–question pairs, we pool their variance estimates and report the corresponding CI half\-width,

hpool=t0\.975,νt0\.975,2​1n​∑i=1nhi2,ν=n⁡\(c−1\)\.h\_\{\\mathrm\{pool\}\}=\\frac\{t\_\{0\.975,\\nu\}\}\{t\_\{0\.975,2\}\}\\sqrt\{\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}h\_\{i\}^\{2\}\},\\qquad\\nu=n\(c\-1\)\.Table[9](https://arxiv.org/html/2609.36253#A5.T9)reports pooled half\-widths across the 18 condition–question pairs for each question and across all 72 pairs, while Table[11](https://arxiv.org/html/2609.36253#A5.T11)reports the individual CI half\-widths\.

Table 9:Run\-to\-run uncertainty across repeated FA runs\. Entries are pooled 95% CI half\-widths,hpoolh\_\{\\mathrm\{pool\}\}, across the condition–question pairs in each row\. The final column reports the mean of the three pairwise Spearman rank correlations in PFS between repeated runs; for “All questions,” correlations are computed after pooling the condition–question observations across the four questions\. Smaller CI half\-widths and larger rank correlations indicate greater stability across repeated elicitation\.Run\-to\-run variation is small relative to differences between model conditions\. In the variance decomposition, the run\-to\-run variance component is2\.9%2\.9\\%of the between\-condition component for PFS, corresponding to an intraclass correlation of0\.9710\.971\. The corresponding ratios are1\.8%1\.8\\%forSstructS\_\{\\mathrm\{struct\}\}and below0\.3%0\.3\\%forSaccS\_\{\\mathrm\{acc\}\},SadaptS\_\{\\mathrm\{adapt\}\}, andScenterS\_\{\\mathrm\{center\}\}\. Thus, accuracy and center alignment are highly stable across repeats, while structure has the largest run\-to\-run variance relative to between\-condition differences among the PFS components\.

The qualitative conclusions are similarly stable\. In 70 of the 72 condition–question pairs, the same component is limiting in all three runs:SstructS\_\{\\mathrm\{struct\}\}in 69 pairs andSadaptS\_\{\\mathrm\{adapt\}\}in one\. This reinforces the main result that structure, rather than accuracy, is typically the limiting dimension of population fidelity\. Among the as\-released conditions, the adaptability ratioAAnever crosses one across runs, so no condition switches between compression and amplification\. Model\-condition rankings are also stable: pairwise Spearman correlations in PFS between runs range from0\.8890\.889to0\.9650\.965across questions\. Finally, 63 of the 72 pairs retain identical cell sets in all three runs, so for most pairs the observed movement reflects sampled responses rather than changes in evaluation support\.

Much of the remaining PFS sensitivity occurs near the clipping boundary forSstructS\_\{\\mathrm\{struct\}\}\. BecauseSstruct=max⁡\(0,ρ\)S\_\{\\mathrm\{struct\}\}=\\max\(0,\\rho\), a small run\-to\-run change that movesρ\\rhoacross zero can move PFS between zero and a positive value\. Nineteen of the 72 condition–question pairs touch this boundary across runs\. Near\-zero PFS values should therefore not be interpreted as fine\-grained rankings: they primarily indicate little or no recovered population structure\.

The paired fine\-tuning effects are also stable across repeated FA runs \(Table[10](https://arxiv.org/html/2609.36253#A5.T10)\)\. Across the 24 model–question comparisons for each cultural target, the run\-specific meanΔ​PFS\\Delta\\mathrm\{PFS\}on all retained cells and the run\-specific meanΔ​Scenter\\Delta S\_\{\\mathrm\{center\}\}in both scopes vary by at most0\.0050\.005\. For population\-wideΔ​PFS\\Delta\\mathrm\{PFS\}, the sign is unchanged across all three runs in 23 of 24ℱde\\mathcal\{F\}\_\{\\text\{de\}\}comparisons and 18 of 24ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}comparisons\. The sign ofΔ​Scenter\\Delta S\_\{\\mathrm\{center\}\}is unchanged in all 24 comparisons for each target and scope\.

Within the target countries, we summarize individual\-comparison run\-to\-run uncertainty by pooling the 24 model–question\-specific variance estimates using the same procedure above \(n=24n=24,ν=48\\nu=48\)\. The resulting pooled 95% CI half\-width is0\.0730\.073forΔ​PFS\\Delta\\mathrm\{PFS\}in Germany and0\.1100\.110in Mexico\. The corresponding pooled half\-widths forΔ​Scenter\\Delta S\_\{\\mathrm\{center\}\}are only0\.0050\.005and0\.0030\.003, respectively\. Thus, center\-alignment shifts are substantially more stable to FA sampling than target\-country PFS changes\. These intervals quantify uncertainty attributable to repeated FA sampling with the survey, prompts, model parameters, and evaluation definitions held fixed; they are not confidence intervals for the mean effect across model–question comparisons\. Appendix[E\.3](https://arxiv.org/html/2609.36253#A5.SS3)separately examines sensitivity to the metric specification\.

Table 10:Paired fine\-tuning effects across the three repeated FA runs\. Each model–question entry reports the mean fine\-tuned\-minus\-as\-released change across the three runs±\\pmits 95% confidence interval, computed on the cells retained in all runs of both conditions\. “All questions” averages the four question\-specific changes within each run before constructing the interval, while “All models” averages the six model\-specific changes within each run\. Positive values indicate increases relative to the corresponding as\-released condition\.Table 11:Run\-to\-run uncertainty by model, question, and condition\. Question\-specific entries are 95% CI half\-widths for the mean score across the three FA runs,t0\.975,c−1​σi/ct\_\{0\.975,\\,c\-1\}\\sigma\_\{i\}/\\sqrt\{c\}\. “All questions” and “All models” entries pool the corresponding run\-level variance estimates using their combined degrees of freedom\.
### E\.2Agreement Across Elicitation Modes

To assess whether the results depend on how responses are elicited, we compare the rankings of the 20 conditions evaluated under both NTP and FA\. Table[12](https://arxiv.org/html/2609.36253#A5.T12)reports the Spearman correlation between modes for the PFS, its components, and center alignment, calculated separately for each question\. Higher correlations indicate that the relative ordering of model conditions is preserved across elicitation methods\.

Table 12:Agreement between NTP and FA elicitation\. Each entry is the Spearman rank correlation between the two modes across the 20 conditions evaluated under both\. The structure column uses the untransformed correlationρ\\rho\.Fine\-tuning effects are also directionally similar across elicitation modes\. NTP and FA agree on the sign ofΔ​PFS\\Delta\\mathrm\{PFS\}in 38 of 48 paired comparisons, or 35 of 41 after excluding comparisons with zero change under either mode\.

### E\.3Metric Sensitivity

The Population Fidelity Score requires several measurement choices, including how between\-group dispersion is summarized, how structure is measured, how the three components are combined, and how cells are weighted\. We test whether the main conclusions depend on these choices by recomputing the analysis under alternative specifications while holding the underlying model responses, retained cells, and nEMD distances fixed\.

##### Alternative specifications\.

We consider seven changes individually and one combined specification\. For adaptability, we replace the median pairwise distance with the mean, or use the one\-sided scoremin⁡\(A,1\)\\min\(A,1\)so that amplification is not penalized\. For structure, we replace Spearman’sρ\\rhowith Pearson’srror Kendall’sτb\\tau\_\{b\}\. Separately, we test an unclipped version of the default Spearman measure that preserves negative correlations rather than mapping them to zero\. We define the corresponding signed score as

PFSsigned=sign⁡\(ρ\)​\(Sacc⋅Sadapt⋅\|ρ\|\)1/3,\\mathrm\{PFS\}\_\{\\mathrm\{signed\}\}=\\operatorname\{sign\}\(\\rho\)\\left\(S\_\{\\mathrm\{acc\}\}\\cdot S\_\{\\mathrm\{adapt\}\}\\cdot\|\\rho\|\\right\)^\{1/3\},wheresign⁡\(ρ\)\\operatorname\{sign\}\(\\rho\)is−1\-1,00, or11according to whetherρ\\rhois negative, zero, or positive\. Thus, the magnitude of the score is computed from\|ρ\|\|\\rho\|, while its sign preserves the direction of the structure correlation\. Unlike the default PFS, this sensitivity variant can take negative values\.

We also replace the geometric mean with the arithmetic mean and, separately, weight accuracy and center alignment by the number of survey respondents in each cell\. The combined specification uses mean pairwise distance, Pearson structure without clipping, and respondent weighting\. Table[13](https://arxiv.org/html/2609.36253#A5.T13)compares these alternatives with the default definition\.

Table 13:Sensitivity of Population Fidelity to alternative metric specifications\. Acc\., Adapt\., Struct\., and PFS report the mean and median across the 164 evaluated combinations\. “Struct\. limit” reports how often structure is the lowest PFS component, first at the population level and then across 4,920 subgroup\-level scores, including the Mixtral reference series\.ρ\\rhovs\. default is the Spearman correlation between PFS under each specification and under the default\. Fine\-tuning columns report mean paired changes in PFS; parentheses give the numbers of increases and decreases\. The final columns report mean changes in PFS and center alignment within the corresponding target country\. For specifications without clipping, the structure value retains its sign and PFS may be negative\.
##### Main conclusions are stable\.

The central results are largely unchanged across specifications\. Accuracy remains substantially higher than structure: mean accuracy ranges only from0\.7730\.773to0\.7750\.775, whereas mean structure ranges from0\.0550\.055to0\.0980\.098\. Structure remains the limiting component in at least 156 of 164 population\-level evaluations and 4,486 of 4,920 subgroup\-level evaluations; accuracy is never limiting\. Among specifications retaining the geometric mean, PFS rankings remain highly similar to the default \(ρ≥0\.979\\rho\\geq 0\.979\)\.

At the population\-wide scope, neither CultureLLM\-style condition improves PFS on average under any specification\. Among geometric\-mean specifications, mean paired changes range from−0\.028\-0\.028to−0\.016\-0\.016for German fine\-tuning and from−0\.033\-0\.033to−0\.012\-0\.012for Mexican fine\-tuning\. Target\-country results are somewhat more specification\-sensitive\. Within Mexico, center alignment improves by approximately0\.0380\.038across specifications, whereas target\-country PFS is negative under all but the unclipped specification, where the mean change is effectively zero\. Within Germany, target\-country PFS is negative under the default and most alternative specifications but becomes positive under the combined specification \(Δ​PFS=\+0\.016\\Delta\\mathrm\{PFS\}=\+0\.016\)\. Thus, the population\-wide conclusion is robust, while the sign of a target\-country mean can depend on the metric specification\.

##### Where the specification matters\.

Some choices affect the numerical scale without changing the main conclusions\. Replacing median with mean pairwise distance raises average adaptability from0\.4960\.496to0\.5210\.521\. Pearson correlation produces somewhat higher structure scores than Spearman, while Kendall correlation produces somewhat lower scores\. Removing the zero clipping converts most nonpositive\-structure cases into negative PFS values rather than zeros\.

The largest change comes from replacing the geometric mean with an arithmetic mean\. Mean PFS rises from0\.2430\.243to0\.4510\.451, all exact zeros disappear, and the ranking correlation with the default falls to0\.6690\.669\. This occurs because the arithmetic mean allows high accuracy and adaptability to compensate for very low structure, whereas the geometric mean treats the three dimensions as jointly required\.

Finally, paired fine\-tuning comparisons are evaluated on the intersection of cells retained by the fine\-tuned and as\-released conditions\. Scoring each condition on its own retained cells produces slightly different changes because response coverage itself then affects adaptability and structure\. Across the sensitivity specifications, this alternative does not change the sign of the pooled mean fine\-tuning effects\. We therefore retain the common\-cell construction so that paired differences reflect changes in model responses rather than changes in evaluation support\.

## Appendix FDistribution\-Matched and Subgroup\-Matched Fine\-Tuning

The CultureLLM\-style conditions used in the main analysis train toward a single target answer per question\. The reduction in between\-group variation reported in Section[6](https://arxiv.org/html/2609.36253#S6)may therefore reflect that objective rather than survey\-based adaptation in general\. To examine whether Population Fidelity distinguishes alternative adaptation objectives, we evaluate two additional methods on a common backbone\. We compare the as\-released model with four adaptation conditions: CultureLLM\-style German fine\-tuning \(ℱde\\mathcal\{F\}\_\{\\text\{de\}\}\), CultureLLM\-style Mexican fine\-tuning \(ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}\), distribution matching \(SimLLCultureDist\), and subgroup matching \(SubPOP\)\.

SimLLCultureDist\([Cao et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib7)\)trains toward aggregate country\-level response distributions\. These targets contain country\-level response distributions but no within\-country demographic subgroup information\. SubPOP\([Suh et al\., 2025](https://arxiv.org/html/2609.36253#bib.bib26)\), by contrast, trains toward response distributions for demographic subgroups and therefore explicitly incorporates subgroup\-level information\. These conditions provide complementary examples of how fine\-tuning approaches with different supervision targets score on the PFS components\.

We fine\-tune Qwen3\-VL\-2B\-Thinking using the released data and training recipes of SimLLCultureDist and SubPOP\. SimLLCultureDist contains 6,841 country\-level response distributions from 46 countries and uses a first\-token KL\-divergence objective\. SubPOP\-Train contains 71,026 subgroup\-level response distributions covering 3,229 American Trends Panel questions and 22 United States subgroups, using the same loss\. We evaluate both conditions using the same procedure as the other model conditions\. Table[14](https://arxiv.org/html/2609.36253#A6.T14)reports first\-run scores; neither condition is included among the 164 combinations analyzed in the main text\.

The training data also differ in their overlap with the evaluation items\. The distribution\-matched data contain the happiness and social\-trust items for four of the five evaluation countries\. SubPOP\-Train contains eight religious\-attendance items and one social\-trust item for United States subgroups only\. Happiness and political ideology are absent from SubPOP\-Train, and neither Germany nor Mexico is represented as a subgroup\-training target\.

Table 14:Population Fidelity diagnostics for Qwen3\-VL\-2B\-Thinking under CultureLLM\-style German fine\-tuning \(ℱde\\mathcal\{F\}\_\{\\text\{de\}\}\), CultureLLM\-style Mexican fine\-tuning \(ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}\), distribution matching \(SimLLCultureDist\), and subgroup matching \(SubPOP\)\. Scores are from the first run on all retained cells \(687 cells; 639 for political ideology\)\.AAis the raw adaptability ratio; PFS uses the corresponding symmetric adaptability scoreSadaptS\_\{\\mathrm\{adapt\}\}\. “Mean change from as released” averages the within\-question difference from the as\-released condition across the four questions\. The final five rows report the median of each condition across its eight question–mode combinations\.The distribution\-matched condition attains the highest median center alignment of the five conditions on this backbone \(0\.8830\.883versus0\.8270\.827–0\.8430\.843\) and improves on the as\-released model in seven of eight question–mode combinations\. PFS gives a more mixed result: it improves in three combinations, declines in three, and remains zero in two, with median PFS increasing from0\.2590\.259to0\.3300\.330\.

Center alignment can also obscure large differences in the PFS components\. For social trust under NTP,ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}and the distribution\-matched condition attain nearly identical center alignment \(0\.8150\.815and0\.8140\.814\), but their PFS values are0\.0000\.000and0\.4250\.425, respectively\.ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}strongly compresses between\-group variation \(A=0\.231A=0\.231\) and recovers no positive structure \(Sstruct=0S\_\{\\mathrm\{struct\}\}=0\), whereas the distribution\-matched condition preserves substantially more variation \(A=0\.818A=0\.818\) and positive structure \(Sstruct=0\.119S\_\{\\mathrm\{struct\}\}=0\.119\)\. Thus, similar center alignment can coexist with very different representations of variation among groups \(Figure[16](https://arxiv.org/html/2609.36253#A6.F16)\)\.

\(a\)Social trust, NTP: component profiles on all cells, German cells, and Mexican cells\.\(b\)Happiness, FA: component profiles on all cells, German cells, and Mexican cells\.\(c\)Social trust, NTP: change from as released, all cells\.\(d\)Happiness, FA: change from as released, all cells\.
Figure 16:Population Fidelity and center alignment for five Qwen3\-VL\-2B\-Thinking conditions: as released,ℱde\\mathcal\{F\}\_\{\\text\{de\}\},ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}, distribution\-matched, and subgroup\-matched\.\(a,b\)Component profiles on all retained cells, German cells, and Mexican cells for social trust under NTP and happiness under FA, respectively\. Diamonds show PFS and hollow squares center alignment\.\(c,d\)Adapted minus as\-released changes on all retained cells for the same two settings\.The subgroup\-matched condition does not consistently recover group structure in this evaluation\. Its median adaptability ratio isA=0\.263A=0\.263, medianSstruct=0\.032S\_\{\\mathrm\{struct\}\}=0\.032, and structure is the limiting component in all eight question–mode combinations\. PFS declines relative to the as\-released model in six of eight combinations\. Its two improvements occur for religious attendance, which has the greatest item\-level overlap with the SubPOP training data among our evaluation questions\. For happiness under FA, for example, the distribution\-matched and subgroup\-matched conditions have nearly identical center alignment \(0\.9530\.953and0\.9520\.952\) but PFS values of0\.4370\.437and0\.2500\.250, respectively, reflecting lower adaptability and structure under subgroup matching\.

These results should be interpreted as transfer across datasets and population definitions: SubPOP is trained on United States demographic subgroups, whereas our evaluation compares demographic cells across five WVS countries\. They therefore show that subgroup\-level distributional training does not automatically transfer to the between\-group structure of these data, rather than that subgroup matching generally fails to recover population structure\.

Because the CultureLLM\-style conditions target individual countries, Figure[16](https://arxiv.org/html/2609.36253#A6.F16)also evaluates all conditions on German and Mexican cells separately\. Within Germany,ℱde\\mathcal\{F\}\_\{\\text\{de\}\}reduces both center alignment and PFS in six of eight combinations and compresses variation among German cells \(medianA=0\.164A=0\.164versus0\.7330\.733for the as\-released model\)\. Within Mexico,ℱmx\\mathcal\{F\}\_\{\\text\{mx\}\}raises center alignment by0\.1030\.103on average, but PFS improves in only two combinations and declines in four; medianAAfalls from0\.7280\.728to0\.4530\.453\.

The additional distribution\-trained conditions likewise show that changes in center alignment do not uniquely determine changes in PFS\. Across the target\-country evaluations, structure remains the limiting component in 77 of 80 combinations\. These experiments involve only one backbone and substantially different training targets and data coverage, so they are intended to demonstrate how PFS distinguishes adaptation profiles rather than to establish comparative performance among fine\-tuning methods\.