Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups

# Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups
Source: [https://arxiv.org/html/2607.27232](https://arxiv.org/html/2607.27232)
1University College London, London, England\.h\.shani\-narkiss@ucl\.ac\.uk2Ben Gurion University of the Negev, Be’er Sheba, Israel\.\{mickyfi,orentsur\}@bgu\.ac\.il†These authors contributed equally to this work\.

###### Abstract

Large Language Models \(LLMs\) are increasingly shaping how we consume information and form our worldview\. This raises concerns beyond bias in AI: do LLMs grasp the emotional nuances conveyed via textual framing? In this work, we empirically evaluate how well an array of LLMs aligns with human emotional perception\. Considering news headlines covering political and geopolitical conflicts, both human participants \(n = 3011, a representative sample of the U\.K\. adult population, via a YouGov survey\) and seven LLMs answered whether headlines evoked sympathy for a specified side in a conflict\. We find that the correlation between AI and human evaluations varies across models, ranging from very high \(ρ=0\.789\\rho=0\.789, GPT\-5\.2\) to medium \(ρ=0\.4\\rho=0\.4,Mistral Large 2512\)\. Crucially, the leading models are broadly aligned with human judgments across all demographic subgroups, including age, gender, level of education, prior geopolitical knowledge, and participants’ predispositions regarding the conflict, although there are statistically significant differences between groups\. This research, with its robust design and large, demographically diverse dataset, offers the most comprehensive evaluation of LLMs’ comprehension of news framing to date\. Findings highlight an important, often\-ignored aspect of differential alignment: even when aggregate performance is high, AI alignment is not universal – it may correspond differently with demographic features and cultural norms\. Considering or ignoring the need for differential alignment may therefore have significant implications for the development of ethical and useful AI systems\.

AI Agents, based on Large Language Models \(LLMs\), are rapidly becoming indispensable mediators of information, profoundly influencing the ways individuals seek information, consume news111”I started asking Perplexity—an A\.I\.\-based system for searching the web—“What’s going on in the world today?” In response, it reliably conjured a short news summary that was informative and unsolicitous, not unlike the section in The Economist headed “The World in Brief\.” ” A\.I\. Is Coming for Culture, The New Yorker \(retrieved 4/25/26\)and formulate worldviews \(?,?,?,?\)\. Considerable attention has focused on the impact of the algorithmic feed on the perception of news and social polarization \(?,?,?,?,?\), on LLMs’ hallucinations, bias and fairness \(?,?,?,?,?\), the models’ ability to distinguish belief from knowledge and facts \(?\), and models’ mis/alignment with human values and goals \(?,?,?,?,?,?\)\. However, another fundamental concern pertains to their alignment with human comprehension of textual nuances, particularly emotional framing\.

Framing, an intrinsic element of compelling writing, layers texts with subtle context, shaping the reader’s perception beyond mere factual content \(?,?\)\. If LLMs fail to process and convey such nuanced framing accurately, their widespread use for content mediation may impair users’ ability to appreciate the richness, context, and intricacies of original texts\. The focus on emotional framing, beyond a mere stab into literary devices, is important: the framing of news may influence public opinion \(?,?, ?,?\), shape public discourse \(?,?\), and ultimately increase social polarization and distrust in social and democratic institutions \(?,?,?\)\. Thus, beyond evaluating raw predictive or linguistic performance of an AI agent over common benchmarks \(?,?\), it is critical to ask: To what degree is an AI Agent’s perception of sympathy and framing aligned with human readers and news consumers? Unfortunately, despite the growing interest in ‘framing’ as a language processing task, the assessment of*alignment*over frames is scarce and based on limited annotation \(?,?,?\)\.

In this work, we present the first comprehensive empirical evaluation of the correspondence between human readers’ perception of emotional framing and an array of LLMs\. A representative sample \(N=3011N=3011\) of the U\.K\. adult population was recruited by YouGov222[https://yougov\.co\.uk/](https://yougov.co.uk/), a leading global polling agency\. Participants answered a series of Yes/No questions related to news headlines covering major political and geopolitical conflicts\. Each question\-headline pair was answered by approximately 500 responders of varying demographics\. Each of the question\-headline pairs was used to prompt seven different LLMs,333ClaudeSonnet4\.5, GPT4, GPT5\.2, GROK4\.1\-Fast, Gemini3\-Flash, MistralLarge2512, and DeepSeek3\.2obtaining each model’s perception of the framing expressed in each text\.

With hundreds of answers per question, provided by human readers and AI models, we can measure the correlation between human and machine perceptions of emotional framing in news headlines\. Leveraging the size of the panel, its rich sociodemographic characteristics, and the number of answers per question \(see SM[1](https://arxiv.org/html/2607.27232#S1)\), we were able to correlate and study the alignment between the model’s comprehension and the comprehension of fine\-grained sociodemographic groups, namely: gender, first language, age, education, NRS \(UK\-social grade\), political awareness, self\-reported prior knowledge and preexisting stance related to the different conflicts\. An illustration of our experimental pipeline, summarizing the experimental setting, is presented in Figure[1](https://arxiv.org/html/2607.27232#S0.F1)\.

![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/yougov_gpt_pipeline_june_26.jpg)Figure 1:Illustration of the research design and the experimental pipeline\.## Method

##### Data

We extracted news headlines from the GDELT Project444GDELT \([https://www\.gdeltproject\.org/](https://www.gdeltproject.org/)\) is a project supported by Google Jigsaw, monitoring and curating “the world’s broadcast, print, and web news”\.\(?\) using a set of predefined keywords related to three major political and geopolitical conflicts: The ongoing Russo\-Ukrainian war \(2022–\), the war in Gaza \(2023\-2025\), and the 2024 U\.S\. Presidential Campaign\. The initial dataset contained∼800​K\\sim 800Kunique headlines, from which a subset of headlines was retained: for each topick=36k=36headlines were sampled at random, and anotherk′=36k^\{\\prime\}=36headlines, deemed “nuanced” \(framing\), were selected\. The final dataset consists of 216 headlines \(72 per conflict\)\. The exact procedure used for headline selection is available in SM[1](https://arxiv.org/html/2607.27232#S1), along with the final list of headlines\.

##### Experimental design

Our experimental design involves the use of two mirrored yes/no questions to assess the sympathetic framing of a given headline:

1. 1\.Does this text create sympathy towards A? <TEXT\>
2. 2\.Does this text create sympathy towards B? <TEXT\>

Combining the binary answers for the mirrored questions, the evaluation of a*headline*can take four values: \(i\) sympathy for neither, \(ii\) sympathy for A, \(iii\) sympathy for B, or \(iv\) sympathy for both A and B\.

While in some cases, a single evaluation for a single headline\-question pair may not be very indicative, obtaining multiple evaluations for each question\-headline pair provides a strong accumulative signal\. SolicitingnnY/Nresponses to a specific questionqiq\_\{i\}, we useri∈\[0,1\]r\_\{i\}\\in\[0,1\]to denote the percentage of positive responses out ofnnforqiq\_\{i\}\. That is,rir\_\{i\}indicates the level555This value can be binarized by checking whetherri<0\.5r\_\{i\}<0\.5orri≥0\.5r\_\{i\}\\geq 0\.5, forming a Bernoulli trial, but keeping the raw value reflects the opaqueness or explicitness of the framing\.of sympathy toward the target of the question \(A or B\)\.

Given a list of questionsq1,…,kq\_\{1,\.\.\.,k\}\(denoteQQ\) and a modelMM, we obtain two corresponding lists of values:RH=r1H,…,rkHR^\{H\}=r^\{H\}\_\{1\},\.\.\.,r^\{H\}\_\{k\}, holding values based on responses solicited from human readers, andRM=r1M,…,rkMR^\{M\}=r^\{M\}\_\{1\},\.\.\.,r^\{M\}\_\{k\}, whereriHr\_\{i\}^\{H\}is the percentage of positive human responses toqiq\_\{i\}, andriMr\_\{i\}^\{M\}is the percentage of positive responses obtained by prompting a modeln=100n=100times with the same questionqiq\_\{i\}\.666Repeated prompting, often referred to as self\-consistency testing, is a common practice necessary due to the stochastic inner\-workings of LLMs \(?\)\.

##### Measuring Alignment

Having two corresponding lists of values,RHR^\{H\}andRMR^\{M\}, we use the Spearman correlationρ​\(RH,RM\)\\rho\(R^\{H\},R^\{M\}\)to quantify the alignment between the human perception and the model’s perception of news articles\. Given the size and the representative nature of panel, we can correlate the model’s perception with the perception of a specific demographicΛ\\Lambdaby considering only responses provided by participants of that demographic\. The alignment between this subgroup and a model is given byρ​\(RHΛ,RM\)\\rho\(R^\{H\_\{\\Lambda\}\},R^\{M\}\)\. Spearman’s rank correlation coefficient, ρ, ranges from −1, indicating a perfect inverse monotonic association, to \+1, indicating a perfect positive monotonic association, with 0 indicating no association\. Following common interpretive conventions, correlations around 0\.4 are typically considered moderate, whereas correlations near 0\.8 indicate strong to very strong alignment \(?,?,?\)\. We test the significance of the results via permutation testing \(?,?,?\), see formal definitions, implementation details, and worked examples in SM[4](https://arxiv.org/html/2607.27232#S4)\.

##### Survey

A panel of participants was recruited by YouGov\. Candidates were given a short explanation about the nature of the survey, after which they could decline participation or opt\-in\. The process conformed to YouGov’s Code of Conduct & Ethics \(?\) and was approved by an Institutional Review Board \(\#SISE\-2024\-48\)\. Opting\-in, participants first answered questions about prior knowledge \(“How much, if anything, would you say you know about each of the following topics?”\) and about their preexisting stance \(“Would you say you have a more favorable view of Side A or Side B, or neither?”\)\. Participants were randomly assigned to one of six experimental groups \(splits\), each group was exposed to 36 headlines \(12 per topic\), for which they were asked to answer the twin yes/no questions described above\. The recruited panel \(N=3011N=3011\) formed a representative sample of the adult UK population, providing rich demographic data, including self\-reported topic knowledge and emotional attachment\. Full details about the recruitment process and the demographic distribution of responders \(overall and within each split\) are available in SM[2](https://arxiv.org/html/2607.27232#S2)\. Headline assignment to splits can be found in SM[1](https://arxiv.org/html/2607.27232#S1)\. Details about the administration of the survey are available in SM[3](https://arxiv.org/html/2607.27232#S3)\.

![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/survey_models_demographic_heatmap.jpeg)

![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/survey_models_demographic_heatmap_2.jpeg)

\(a\)Spearman correlation coefficients between selected demographic groups and each of the models\.
![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/survey_gpt.52.demographic_correlations.jpeg)\(b\)Spearman correlation coefficients between selected sub\-demographic groups for ChatGPT 5\.2\.

Figure 2:Alignment of various AI models with survey all responders and specific sub\-demographics\.Social Gradecategories are defined by regards the UK National Readership Survey \[NRS\]\.

## Results

##### Models’ Alignment with the General Population

The main results are presented in Figure[2\(a\)](https://arxiv.org/html/2607.27232#Sx1.F2.sf1)\. All models showed high and statistically significant alignment with human judgments of emotional framing\. However, the level of alignment varied substantially between models and demographic groups\. The heat maps \(Figure 2A\) show a consistent hierarchy of models over demographic and attitudinal subgroups: GPT\-5\.2 performed best overall, with Spearman’sρ=0\.79\\rho=0\.79across all respondents, followed by Grok \(ρ=0\.74\\rho=0\.74\) and GPT\-4 \(ρ=0\.71\\rho=0\.71\)\. Gemini and DeepSeek showed intermediate alignment, with overall correlations ofρ=0\.67\\rho=0\.67andρ=0\.66\\rho=0\.66, respectively\. Claude achieved weaker alignment \(ρ=0\.58\\rho=0\.58\), and Mistral was the least aligned model \(ρ=0\.41\\rho=0\.41\)\.

The ordering of the models was broadly stable across demographic subgroups\. GPT\-5\.2 achieved the highest alignment across most demographic and attitudinal categories, typically reachingρ=0\.75​–​0\.79\\rho=0\.75\\textendash 0\.79, dropping to 0\.7\-0\.71 for older responders and responders with no formal education\. Grok and GPT\-4 followed a similar pattern at slightly lower levels, whereas Gemini and DeepSeek generally occupied the middle range\. Claude and Mistral showed lower correlations across most subgroups\.

##### Alignment over Topics

Alignment levels also varied across topics\. However, some models were more stable than others: the topical alignment levels of GPT\-5\.2 varied between 0\.74 \(Trump\-Harris\) to 0\.79 \(Russia\-Ukrain\) while the topical alignment of Grok ranged from 0\.62 \(Trump\-Harris\) to 0\.77 \(Israel\-Palestine\)\. The poor overall alignment of Mistral \(ρ=0\.41\\rho=0\.41\) are the result of the lack of alignment over Israel\-Palestine \(0\.09\), as it achieved alignment of 0\.59 and 0\.52 over Russia\-Ukrain and Trump\-Harris, respectively\. Full results by topical breakdown are available in SM[5](https://arxiv.org/html/2607.27232#S5)\)\.

##### Alignment Variations across Sub\-demographic Groups

Shifting the focus to the variance in alignment with different sub\-demographics, we focus on GPT\-5\.2, the best performing model \(alignment levels are presented in Figure[2\(b\)](https://arxiv.org/html/2607.27232#Sx1.F2.sf2); detailed results are available in Tables[2](https://arxiv.org/html/2607.27232#S4.T2)and[3](https://arxiv.org/html/2607.27232#S4.T3)in SM[4](https://arxiv.org/html/2607.27232#S4)\)\. No statistically significant difference was found between male and female respondents \(ρ=0\.784​and​0\.783\\rho=0\.784~\\text\{and\}~0\.783, respectively\)\. A statistically significant, though small, difference was found between speakers of English as First Language \(L1\) and non\-native speakers \(ρL​1=0\.790\\rho\_\{L1\}=0\.790vs\.ρL​2=0\.749\\rho\_\{L2\}=0\.749;p=0\.006p=0\.006\)\.

Age and education showed clearer subgroup differences\. The alignment was highest with respondents aged 35–44 \(ρ=0\.790\\rho=0\.790\) and lowest among respondents aged 75\+ \(ρ=0\.711\\rho=0\.711\), a significant max–min \(range\) difference withp=0\.001p=0\.001\)\. The 18–24 versus 75\+ comparison was also significant after correction\. For education, alignment was lowest among respondents with no formal qualifications \(ρ=0\.698\\rho=0\.698\) and higher among those with secondary, undergraduate, or postgraduate education \(ρ=0\.781−0\.790\\rho=0\.781\-0\.790\)\. The contrast between no formal qualifications and secondary education was significant, as was the contrast between no formal qualifications and Masters/PhD respondents\.

UK\-social grade, defined by the National Readership Survey \(NRS\) is commonly used to assess and explain differences in health, media consumption, and political discourse within the UK population \(?,?,?\)\. We find that grade exhibit only limited variation in alignment level: the largest difference was between grades B and E \(ρ=0\.790\\rho=0\.790versusρ=0\.754\\rho=0\.754;p=0\.040p=0\.040\), while adjacent and extreme comparisons were not significant after correction\.

Statistically significant differences in alignment levels between subgroups of different political awareness, knowledge, and attitude \(predesposition\) were also observed\. GPT\-5\.2 was more aligned with respondents reporting high political awareness than with those reporting low awareness \(ρ=0\.792\\rho=0\.792vs\.ρ=0\.76\\rho=0\.76; correctedp=0\.001p=0\.001\)\. Topic\-specific knowledge showed a similar pattern: respondents who “knew nothing” about the topic had lower alignment \(ρ=0\.722\\rho=0\.722\) than those who know “something” or know “a great deal” \(ρ=0\.79\\rho=0\.79\)\. Finally, alignment was lower among respondents with no strong views on the conflict \(ρ=0\.764\\rho=0\.764\) than among those with a slight or strong lean \(ρ=0\.79\\rho=0\.79\)\.

Together, these results show that leading LLMs closely track human judgments of emotional framing across diverse demographic and attitudinal groups\. At the same time, alignment is not perfectly uniform: it is reduced among older respondents, respondents with no formal qualifications, those with lower political or topic\-specific knowledge, and those without strong views\.

## Discussion

In the experimental setting described above, we use a number of LLMs as “readers”\. It is important to distinguish between an LLM’s capacity to perceive framing and its ability to generate content reflecting a specific perspective, e\.g\., \(?,?\)\.

Some studies explore how framing is manifested in AI\-generated news content, revealing that LLMs tend to exhibit more pronounced framing than human authors, particularly in politically and socially sensitive contexts \(?\), inline with studies addressing LLM bias \(?, ?, ?, ?, ?\)\. It is also important to distinguish between the use of an LLM as a classifier of explicit forms of sentiment, stance or even emotions, and their performance over more nuanced categories such as framing \(?\)\.

##### Practical Implication

We find that some models are well aligned with human reading of sympathetic framing of news stories, while other models are poorly aligned\.

However, while the leading models perform generally well across all demographic subgroups and topics, they do exhibit varying levels of alignment across sociodemographic groups and topics\. These results highlight the potential of AI\-based surveys to complement classical human\-based surveys with synthetic polling\.

These observations offer another perspective on the important discussion about alignment and AI ethics, suggesting that differential alignment protocols – aligning models differently catering to different demographics and cultural norms – may be needed\.

It is important to note that differential alignment should be considered only with great caution as demographic alignment bears significant ethical implications: Aligning a model inappropriately \(e\.g\., a non\-transparent decision on the “appropriate” alignment profile made by the model developers\) introduces an inherent yet elusive bias that influences the user’s perception of the news\. Furthermore, given models’ penchant for sycophancy \(?,?\) – generating content that reinforces the user’s perceived views \(?,?\), and the human comfort with echo\-chambers, may result in increased polarization and loss of trust \(?\)\.

##### Limitations

New LLMs and new versions of existing LLMs are released frequently\. At the same time, results vary across models and versions \(e\.g\., GPT4 and GPT5\.2 in Figure[2](https://arxiv.org/html/2607.27232#Sx1.F2)a\)\. We note that in this work, we introduced a sound methodology that could be applied in testing the alignment with any black\-box LLM\. Moreover, the unique survey data we release can serve as a benchmark for future studies\.

Mistral and DeepSeek exhibit a unique trend as they are better aligned with older demographics\. These results, suggesting that different models are based on different practices in training and post\-training tuning, yet again highlight the need for transparency and disclosure\.

Finally, interaction effects between demographics and content \(e\.g\., age and education or topic awareness\) are probably at play and are not addressed in this work\. However, the partial correlations we report are sufficient to demonstrate significant differences in alignment between models and demographic groups\.

## References and Notes

## 1Survey Questions

##### Headline Retrieval from GDELT\.

The Global Database of Events, Language, and Tone \(GDELT777[https://www\.gdeltproject\.org/](https://www.gdeltproject.org/)\) is a comprehensive open\-access project that continuously monitors global news media outlets, automatically extracting and codifying reports of events, key actors, locations, and the tone of coverage in over 100 languages \(?\)\. By providing structured data on a myriad of socio\-political events and related news content, GDELT enables large\-scale, quantitative studies of worldwide media trends\.

A fundamental analytic technique for exploring GDELT data is keyword search\. Researchers can use targeted keywords, phrases, or Boolean expressions to identify and isolate news articles, events, or themes of interest within the dataset\. This facilitates topic\-based filtering, temporal or geographic trend mapping, and content analysis at an unprecedented scale\. The unique coverage supports research event detection, sentiment analysis in multilingual news media, the spread of misinformation, analyzing crisis communication, news framing, climate change recognition, and modeling information diffusion, e\.g\., \(?,?,?,?,?\)\.

##### Data Collection\.

News headlines were aggregated from a range of mainstream outlets via GDELT feeds and curated repositories to ensure broad representation of political leanings and rhetorical styles\. Initial data collection focused on three subjects: \(1\) the 2024 U\.S\. Presidential race, \(2\) the Israel\-Gaza war \(2023\), and \(3\) the Russia\-Ukraine war\. For each subject, headlines were collected within specific temporal windows \(U\.S\. Presidential race: 15/6/2024–15/9/2024; Israel\-Gaza war: 7/10/2023–7/2/2024; Russia\-Ukraine war: 24/2/2022–24/6/2022\) using relevant keywords or letter sequences \(U\.S\. Presidential race: “Kamala”, “Harris”, “Trump”; Israel\-Gaza war: “Israel”, “Gaza”, “Palestin”, “Antisem”, “Hamas”, “IDF”; Russia\-Ukraine war: “Russia”, “Ukraine”, “Kyiv”, “Kiev”, “Zelenskyy”, “Donbas”, “Luhansk”, “Putin”\)\. This process produced 1,672,909 headlines \(U\.S\. Presidential race: 414,379; Israel\-Gaza war: 713,801; Russia\-Ukraine war: 544,729\)\. After removing duplicates, the dataset comprised 824,876 unique headlines \(U\.S\. Presidential race: 199,728; Israel\-Gaza war: 343,322; Russia\-Ukraine war: 281,826\)\.

##### Survey design and headline selection\.

The survey was designed to balance topic coverage and cognitive load, allowing sound statistical analysis while maintaining a reasonable completion time that caps cognitive load and ensures response quality\. To approximate real\-world exposure, we included a set of randomly sampled news headlines\. However, since most headlines in the source datasets were either strictly factual \(e\.g\., ”Russia\-Ukraine war: What happened today”\) or strongly one\-sided \(e\.g\., ”Fears for Mariupol defenders after surrender to Russia”\), we supplemented these with a curated set of more nuanced items \(e\.g\., ”Russia rules out any threat to Ukrainian civilians during military action”\)\. This approach enabled the inclusion of less trivial cases while preserving representativeness\.

The two\-stage headline selection was done as follows: First, 108 headlines were randomly sampled from the compiled datasets\. Second, to identify nuanced items, the full dataset was processed using GPT\-4 with the standard prompt to assess whether headlines elicited sympathy towards one or more sides of a conflict\. Most headlines did not elicit sympathy; among those who did \(∼25%\\sim 25\\%\), the vast majority \(≥95%\\geq 95\\%\) elicited sympathy toward one side only\. To enrich the dataset with more complex cases, we implemented a targeted selection procedure\. Headlines that produced at least one positive response in the initial screening were retained and re\-evaluated across 10 independent runs using the same model and prompt\. From this subset, 108 headlines were manually selected based on either \(i\) consistent elicitation of sympathy toward both sides or \(ii\) variability in responses across the ten runs\. The final corpus consisted of 216 headlines: 108 randomly sampled and 108 selectively curated, a total of 72 headlines per domain \(36 random, 36 selected\)\. In addition, four control headlines per domain \(two with overtly sympathetic framing and two completely unrelated to the topic, see Figure[5\(a\)](https://arxiv.org/html/2607.27232#S3.F5.sf1)\) were included to support participant quality control\. These control items were used by YouGov as an exclusion criteria\.

All headlines were manually verified for authenticity and edited to remove problematic features \(e\.g\., non\-English characters, mixed topics\)

An overview of the selection procedure is provided in Table[1](https://arxiv.org/html/2607.27232#S1.T1)\. The final instrument comprised 432 questions based on 216 headlines\. Participants were assigned to one of six groups \(*Splits*\), each of which evaluated a batch of 36 headlines \(72 questions\)\. The full set of headlines, including source information and data splits, is available in an external repository[\(Google Sheet\)](https://docs.google.com/spreadsheets/d/1HU4BSkdFZ4GUfE5FQc92GYJYYmX-hSMbL97LKLO7d10/edit?usp=sharing)\.

TopicTime SpanKeywords\# matchesUniqueFinal setU\.S\. 2024 Presidential race15/6/2024 – 15/9/2024Kamala, Harris, Trump414,379199,72872Israel\-Gaza war07/10/2023 – 07/02/2024Israel, Gaza, Palestin, Antisem, Hamas, IDF713,801343,32272Russo\-Ukraine war24/02/2022 – 24/06/2022Russia, Ukraine, Kyiv, Kiev, Zelenskyy, Donbas, Luhansk, Putin544,729281,82672Table 1:Overview of the headlines selection process\. The Final Set includes 36 randomly sampled headlines per topic and 36 headlines sampled from the set of “nuanced” headlines\.

## 2Panel Demography

![Refer to caption](https://arxiv.org/html/2607.27232v1/x1.png)Figure 3:Demographic distributions of the study sample\. Subplots show \(A\) Gender, \(B\) Education, and \(C\) Age group classifications\.A representative sample of the U\.K\. adult population was recruited by YouGov888[https://yougov\.co\.uk/](https://yougov.co.uk/), a leading global polling agency\. Candidates were given a brief explanation of the survey’s nature, after which they could decline to participate or opt in\. The process conformed to YouGov’s Code of Conduct & Ethics \(?\) and was approved by an Institutional Review Board \(\#SISE\-2024\-48\)\. Some of the responders \(m=1,308m=1,308\) did not complete the survey or proved inattentive \(failed attention questions that popped up during the survey, see details in SM[3](https://arxiv.org/html/2607.27232#S3)\), and were consequently excluded by YouGov’s\. In total, valid responses were collected fromn=3,011n=3,011responders\. The demographic breakdown of this group is provided below\.

## 3Survey Administration

Participants completed the survey through a web application maintained and administered by YouGov\. Upon opting in, participants were randomly allocated to one of the six groups and first answered general questions about their knowledge of the topics and their predisposition towards the relevant parties \(see Figures[4\(a\)](https://arxiv.org/html/2607.27232#S3.F4.sf1)and[4\(b\)](https://arxiv.org/html/2607.27232#S3.F4.sf2)\)\.

To ensure quality, each survey included a control question placed among the other questions \(see Figure[5\(a\)](https://arxiv.org/html/2607.27232#S3.F5.sf1)\)\. The answers to these questions were not processed with the other questions\. Rather, participants who failed to answer the control questions correctly were disqualified, and their answers were ignored\.

An example of a headline regarding the Presidential campaign is presented in Figure[5\(b\)](https://arxiv.org/html/2607.27232#S3.F5.sf2), and an example of a headline potentially invoking sympathy for both sides of the Russo\-Ukraine war is presented in Figure[5\(c\)](https://arxiv.org/html/2607.27232#S3.F5.sf3)\.

![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/sm/survey_knowledge.png)\(a\)Participants mark their self\-assessed knowledge of the topics they will be asked about\.
![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/sm/survey_fav_US.png)\(b\)Participants indicate their predisposition regarding the parties involved in the questions they will be presented with\.

Figure 4:Knowledge of the topics and predisposition towards the parties involved![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/sm/survey_control.png)\(a\)An example of a control question \- the text is unrelated to the questions\. Participants are expected to answerNoon both questions\.
![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/sm/survey_debate.png)\(b\)Question about a headline reporting on the Presidential debate of September 10th 2024\.
![Refer to caption](https://arxiv.org/html/2607.27232v1/figs/sm/survey_couple.png)\(c\)Question about a headline evoking sympathy to both sides in the Russia\-Ukraine war\.

Figure 5:Examples of survey questions\.
## 4Significance Testing

The main contribution of this work is the evaluation of the alignment of various AI models with specific demographics\. Given two corresponding lists of values,RHR^\{H\}andRMR^\{M\}, we use the Spearman correlationρ​\(RH,RM\)\\rho\(R^\{H\},R^\{M\}\)to quantify the alignment between human and model perceptions of news articles\. Given the size and representative nature of the panel, we can correlate the model’s perception with the perception of a specific demographicΛ\\Lambdaby considering only the responses provided by participants of that demographic\. The alignment of a demographic subgroupΛ\\Lambdawith modelMM, is the Spearman rank correlation:

ρΛ=ρSpearman​\(RΛ,RM\)\\rho\_\{\\Lambda\}=\\rho\_\{\\text\{Spearman\}\}\(R^\{\\Lambda\},R^\{M\}\)
Considering two mutually exclusive demographic subgroupsΛ\\LambdaandΨ\\Psi, we denote the difference between the level of alignment of a model with demographicΛ\\Lambdaand its alignment with demographicΨ\\Psiby:

ΔΛ,Ψ=\|ρΛ−ρΨ\|\\Delta\_\{\\Lambda,\\Psi\}=\|\\rho\_\{\\Lambda\}\-\\rho\_\{\\Psi\}\|
We use permutation testing \(?, ?, ?\) to test whetherΔΛ,Ψ\\Delta\_\{\\Lambda,\\Psi\}is statistically significant\.

In the remainder of this section, we provide the formal definitions and specify the different realizations of the permutation testing\.

We begin by introducing the formal notation and the general form of the permutation test, followed by details on the specific realization used with respect to different sub\-demographic breakdowns: binary, ordinal, and categorical\. A worked example is offered with each type of realization\. We conclude this Appendix with tables that provide comprehensive descriptions and results for all significance tests of differences in alignment levels between ChatGPT5\.2 \(the highest overall alignment\) and different sub\-demographics\.

### 4\.1Formal Notation

##### Questions and Splits

Let\{qi\}i=1k=Q\\\{q\_\{i\}\\\}\_\{i=1\}^\{k\}=Qa set ofkkyes/no questions \(k=432k=432\)\.

The questions inQQare partitioned intoSSdisjoint subsets \(*Splits*\) of equal size \(in practice,\|S\|=6\|S\|=6, see Survey design and headline selection in SM[1](https://arxiv.org/html/2607.27232#S1)\)\.QsQ^\{s\}denotes a specific subset of questions included in thes​\-ths\\text\{\-th\}split \(s∈Ss\\in S\), and useqisq^\{s\}\_\{i\}ifqi∈Qsq\_\{i\}\\in Q^\{s\}\.

##### Responders, Demograpics, and Splits

LetU=\{uj\}j=1nU=\\\{u\_\{j\}\\\}\_\{j=1\}^\{n\}be a group ofnnresponders \(n=3011n=3011\)\.

Each responder is assigned a splitssand only answers questions inQsQ^\{s\}\. We useujsu^\{s\}\_\{j\}to denote the responder’s assigned split\.

Responders can also be characterized by demographic traits\. We useujΛu^\{\\Lambda\}\_\{j\}to denote thatuju\_\{j\}belongs to a specific sub\-demographicΛ\\Lambda\. We useujs,Λu^\{s,\\Lambda\}\_\{j\}to indicate the responder’s split*and*demographic group\. Similarly, we useUs,ΛU^\{s,\\Lambda\}to indicate all responders of demographicΛ\\Lambdain splitss\.

Taking*all*responders into account, each questionqiq\_\{i\}was answered byn/Sn/Sresponders\. However, when considering only responders in specific demographicΛ\\Lambda, questions across different splits may garner a different number of answers due to the distribution of demographic traits and the random assignment of responders to splits\. For example, considering only Male responders \(Λ=Male\\Lambda=\\text\{Male\}\) it may be the case that\|Us,Λ\|≠\|Us′,Λ\|\|U^\{s,\\Lambda\}\|\\neq\|U^\{s^\{\\prime\},\\Lambda\}\|, thus a questionq∈Qsq\\in Q^\{s\}gets a different number of answers by male responders than a questionq′∈Qs′q^\{\\prime\}\\in Q^\{s^\{\\prime\}\}\.

##### Survey results and Positive response ratio

We useRH=r1H,…,rkHR^\{H\}=r^\{H\}\_\{1\},\.\.\.,r^\{H\}\_\{k\}to indicate the list of ratios of positive answers received for questionqiq\_\{i\}\. That is, ifqiq\_\{i\}is answered bymim\_\{i\}responders, out of whichyi≤miy\_\{i\}\\leq m\_\{i\}answered positively,ri=yimir\_\{i\}=\\frac\{y\_\{i\}\}\{m\_\{i\}\}\.

Considering only responders of a specific demographicΛ\\Lambda, we defineRΛ=r1Λ,…,rkΛR^\{\\Lambda\}=r^\{\\Lambda\}\_\{1\},\.\.\.,r^\{\\Lambda\}\_\{k\}, whereMiΛM^\{\\Lambda\}\_\{i\}andyiΛy^\{\\Lambda\}\_\{i\}are the number of responders and positive responses of members ofΛ\\Lambdatoqiq\_\{i\}\. Given thatqi∈Qsq\_\{i\}\\in Q^\{s\}we haveriΛ=y​ΛimiΛr^\{\\Lambda\}\_\{i\}=\\frac\{y\\Lambda\_\{i\}\}\{m^\{\\Lambda\}\_\{i\}\}andmiΛ=\|Us,Λ\|m^\{\\Lambda\}\_\{i\}=\|U^\{s,\\Lambda\}\|\.

RM=\{riM,…,rkM\}R^\{M\}=\\\{r^\{M\}\_\{i\},\.\.\.,r^\{M\}\_\{k\}\\\}denotes the positive ratio returned by modelMMtoq1,…,qkq\_\{1\},\.\.\.,q\_\{k\}\. Given each prompt is issuednntimes,riMr^\{M\}\_\{i\}is the number of positive answers returned toqiq\_\{i\}, divided bynn\.

##### Measuring Alignment

The alignment of a demographic subgroupΛ\\Lambdawith modelMM, is the Spearman rank correlation:

ρΛ=ρSpearman​\(RΛ,RM\)\\rho\_\{\\Lambda\}=\\rho\_\{\\text\{Spearman\}\}\(R^\{\\Lambda\},R^\{M\}\)
Given two mutually exclusive demographic subgroupsΛ\\LambdaandΨ\\Psi, we define

ΔΛ,Ψ=\|ρΛ−ρΨ\|\\Delta\_\{\\Lambda,\\Psi\}=\|\\rho\_\{\\Lambda\}\-\\rho\_\{\\Psi\}\|to indicate the absolute difference between a models’s alignment with demographicΛ\\Lambdaand demographicΨ\\Psi\. We are interested in testing whetherΔΛ,Ψ\\Delta\_\{\\Lambda,\\Psi\}is significant\. Significance is evaluated through permutation testing\.

##### Permutation testing – Design Principles

Three principles apply to every realization of a permutation test used in this paper\.

1. 1\.Per\-participant relabelling:The unit being re\-labelled is the respondentuju\_\{j\}, rather than an isolated response to a questionqiq\_\{i\}; under any permutationπ\\pi, all\|Qs\|=72\|Q^\{s\}\|=72responses provided byujsu^\{s\}\_\{j\}are re\-assigned together\. This preserves within\-respondent answer dependence and is therefore conservative relative to a per\-question shuffle\.
2. 2\.Stratification by Split:Permutations are constrained to preserve, within every Splits∈\{1,…,S\}s\\in\\\{1,\\dots,S\\\}, the number of respondents at each subgroup level\. This matches the experimental design \(Split is fixed by the protocol; only the human group label is exchangeable conditional on Split\) and rules out chance Split×\\timeslabel imbalance\.
3. 3\.FixedRMR^\{M\}vector:RMR^\{M\}is held fixed across all permutations\. The test asks only whether the human side could have produced the observed alignment under random labeling; the model provides the reference axis\.

##### Permutation testing – General form

Given a setΓ\\Gammaof mutually exclusive demographic sub/groups, letπ\\pibe a random partitioning function defined as:

π:U→\{Us,γ¯\}s,γ∈S×Γ\\pi:U\\rightarrow\\\{\\overline\{U^\{s,\\gamma\}\}\\\}\_\{s,\\gamma\\in S\\times\\Gamma\}\(1\)
S\.t\. the design principles stated above:

⋃s,γ∈S×ΓUs,γ¯=U​;​⋂s,γ∈S×ΓUs,γ¯=ϕ​; and​\|Us,γ¯\|=\|Us,γ\|​∀s,γ∈S×Γ\\bigcup\_\{s,\\gamma\\in S\\times\\Gamma\}\\overline\{U^\{s,\\gamma\}\}=U\\text\{~;~\}\\bigcap\_\{s,\\gamma\\in S\\times\\Gamma\}\\overline\{U^\{s,\\gamma\}\}=\\phi\\text\{~; and ~\}\|\\overline\{U^\{s,\\gamma\}\}\|=\|U^\{s,\\gamma\}\|~~\\forall\_\{s,\\gamma\\in S\\times\\Gamma\}
For any two sub\-groupsΛ,Ψ∈Γ\\Lambda,\\Psi\\in\\Gamma, we can now computeRΛ¯\\overline\{R^\{\\Lambda\}\}andRΨ¯\\overline\{R^\{\\Psi\}\}, findρΛ¯\\overline\{\\rho\_\{\\Lambda\}\}andρΨ¯\\overline\{\\rho\_\{\\Psi\}\}, and define the absulute difference:

ΔΛ,Ψ¯=\|ρΛ¯−ρΨ¯\|\\overline\{\\Delta\_\{\\Lambda,\\Psi\}\}=\|\\overline\{\\rho\_\{\\Lambda\}\}\-\\overline\{\\rho\_\{\\Psi\}\}\|
The two\-tailedpp\-value is therefore given by:

pΛ,Ψ=∑1zI​\(Λ,Ψ,π\)zp\_\{\\Lambda,\\Psi\}=\\frac\{\\sum\_\{1\}^\{z\}\\mathrm\{I\}\(\\Lambda,\\Psi,\\pi\)\}\{z\}\(2\)
wherezzis the number of permutations \(we usez=105z=10^\{5\}\) and

I​\(Λ,Ψ,π\)=\{1if​ΔΛ,Ψ≤ΔΛ,Ψ¯0otherwise\\mathrm\{I\}\(\\Lambda,\\Psi,\\pi\)=\\begin\{cases\}1&\\text\{if \}\\Delta\_\{\\Lambda,\\Psi\}\\leq\\overline\{\\Delta\_\{\\Lambda,\\Psi\}\}\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(3\)

### 4\.2Binary Attributes \(Gender, First Language\)

The respondents included in the panel gathered by YouGov reported gender in a binary way:*Male*and*Female*\. Similarly, since all texts were taken from English media outlets and the survey was administered in English, we treat native language as a binary variable, taking the values*English*and*non\-English*\.

In the binary case, the permutation test is realized in a straightforward way – the two\-tailedpp\-value is given by the general form in Equation[2](https://arxiv.org/html/2607.27232#S4.E2)\. However, in the binary case, the two sub\-demographics fully coverΓ\\Gamma:

pΛ,Ψ=∑1zI​\(Λ,Ψ,π\)zp\_\{\\Lambda,\\Psi\}=\\frac\{\\sum\_\{1\}^\{z\}\\mathrm\{I\}\(\\Lambda,\\Psi,\\pi\)\}\{z\}
Worked Example — Binary Case: Gender \(Female vs\. Male\)Setup:The Gender attribute has two groups\. GroupΛ\\Lambda= Female \(\|UΛ\|=1,769\|U^\{\\Lambda\}\|=1,769\) and groupΨ\\Psi= Male \(\|UΨ\|=1,242\|U^\{\\Psi\}\|=1,242\), for a pooled sample of\|UΛ​⋃UΨ\|=3,011\|U^\{\\Lambda\}\\bigcup U^\{\\Psi\}\|=3,011respondents\. The per\-Split cell counts are:Observed statistics\.Computing yes\-rate vectorsRFemaleR^\{\\text\{Female\}\}andRMaleR^\{\\text\{Male\}\}overk=432k=432questions and correlating each withRMR^\{M\}\(in this and subsequent examplesMMis ChatGPT5\.2\):ρFemale=0\.7834,ρMale=0\.7841,ΔFemale,Male=ρFemale−ρMale=−0\.0007\.\\rho\_\{\\text\{Female\}\}=0\.7834,\\quad\\rho\_\{\\text\{Male\}\}=0\.7841,\\quad\\Delta\_\{\\text\{Female,Male\}\}=\\rho\_\{\\text\{Female\}\}\-\\rho\_\{\\text\{Male\}\}=\-0\.0007\.One permutation step \(illustration\):In Split 1 we have 301 Females and 216 Males\. We draw a random permutation of 517 labels \(301 F’s and 216 M’s\) within Split 1 independently, and similarly for each of the other five Splits\. The per\-Split F/M counts are identical to the observed counts – only the*identities*of which respondents are labelled Female vs\. Male may change\. Thus, ifuiu\_\{i\}, originally inUs=1,F​e​m​a​l​eU^\{s=1,Female\}was reassigned byπ\\pitoUs=1,M​a​l​e¯\\overline\{U^\{s=1,Male\}\}, all of her7272answers are now considered to be provided by a male responder\.Result afterz=105z=10^\{5\}permutations:90,23890,238permutations producedI​\(∘\)=1\\mathrm\{I\}\(\\circ\)=1, givingpFemale,Male=90,238100,000≈0\.902\(ns\)\.p\_\{\\text\{Female,Male\}\}=\\frac\{90,238\}\{100,000\}\\approx 0\.902\\quad\(\\text\{ns\}\)\.Interpretation:Both groups align equally with GPT\-5\.2; there is no statistically significant gender gap\.

#### 4\.2\.1Multi\-Category*Ordinal*Attributes \(Age, Education, Social Grade, General Political Awareness\)

Some demographic attributes have ordinal sub\-categories\. Typical examples are age and education\. Similarly, self reported levels of general political awareness, the level of knowledge of a specific conflict or the predisposition towards a specific side in a specific conflict are ordinal \(see Figure[4](https://arxiv.org/html/2607.27232#S3.F4)in Appendix[2](https://arxiv.org/html/2607.27232#S2)\)\.

##### Family of named pair comparisons

Givenccordinal sub\-categories of a demographic traitΓ\\Gamma\(\|Γ\|=c≥3\|\\Gamma\|=c\\geq 3\), we test the significance ofΔΛ,Ψ\\Delta\_\{\\Lambda,\\Psi\}in a family ofc\+1c\+1configurations of\(Λ,Ψ\)\(\\Lambda,\\Psi\)pairs \(?\):

\(Λ=i,Ψ=i\+1\)​∀i∈\{1,…,c−1\}\(\\Lambda=i,\\Psi=i\+1\)~~\\forall i\\in\\\{1,\.\.\.,c\-1\\\}c−1c\-1ordinal*adjacent*pairs

\(Λ=1,Ψ=c\)\(\\Lambda=1,\\Psi=c\)the two sub\-categories at the*extremes*

\(Λ=arg⁡mini⁡ρi,Ψ=arg⁡maxi⁡ρi\)sub\-categories ofmin/maxalignment\\hskip 14\.0pt\(\\Lambda=\\arg\\min\_\{i\}\\rho\_\{i\},\\Psi=\\arg\\max\_\{i\}\\rho\_\{i\}\)~~~~~~~~\\text\{sub\-categories of \\emph\{min/max\} alignment\}

##### Marginal and family\-wisepp\-values:

The marginal \(raw\)pp\-value for pair\(Λ,Ψ\)\(\\Lambda,\\Psi\)is given by the general form in Equation[2](https://arxiv.org/html/2607.27232#S4.E2)\. However, the*adjacent*and*extreme*pairs share respondents and therefore marginalpp\-values are dependent Bonferroni and Benjamini\-Hochberg corrections are suboptimal\. We thus use the exact*maximum\-statistic*family\-wise null distribution: for*each*permutationπ\\piand adjacent pair of sub\-categoriesΛ\\LambdaandΨ\\Psiwe replaceΔΛ,Ψ¯\\overline\{\\Delta\_\{\\Lambda,\\Psi\}\}in Equation[3](https://arxiv.org/html/2607.27232#S4.E3), withΔγ,ψ¯\\overline\{\\Delta\_\{\\gamma,\\psi\}\}, where

γ=arg⁡mini⁡ρi​and​ψ=arg⁡maxi⁡ρi\\gamma=\\arg\\min\_\{i\}\\rho\_\{i\}~\\text\{and\}~\\psi=\\arg\\max\_\{i\}\\rho\_\{i\}\. Thepp\-value based on this maximal Family\-Wise Error Rate statistic is denoted by:

pΛ,ΨFWERp\_\{\\Lambda,\\Psi\}^\{\\text\{FWER\}\}\.

##### Maximum\-vs\-minimum correlation \(range test\):

Note that while the\(Λ,Ψ\)\(\\Lambda,\\Psi\)pairs in the*adjacent*and*extreme*test are pre\-specified, the pair in the*min/max*test is unknown in advance\. Therefore inΔΛ,Ψ\\Delta\_\{\\Lambda,\\Psi\}we use the two subclasses producing the largest observed delta \(by definition of*min/max*test\) and useΔγ,ψ¯\\overline\{\\Delta\_\{\\gamma,\\psi\}\}as specified above – the sub\-demographics producing the minimal and maximal alignment in each permutation\.prangep^\{\\text\{range\}\}denotes thatpp\-value obtained for the range text\.

Worked Example – Multi\-Category Case: Agec=7c=7##### Setup\. The Age attribute hasc=7c=7ordered levels\. Group sizes and observed Spearman correlations with GPT\-5\.2 are: ##### One permutation step \(illustration\)\. Consider Split 1, which has 517 respondents\. Their age\-group labels form a vector of 517 values in\{1,…,7\}\\\{1,\\dots,7\\\}with fixed per\-level counts \(e\.g\., 91 respondents in 35–44 and 26 in 75\+ within Split 1\)\.π\\pidraws a uniformly random permutation of these 517 labels — keeping the counts for each of the 7 age groups exactly as observed — and assigns the shuffled labels back, keeping the same number of age labels as in each original split \(\|Us,γ¯\|=\|Us,γ\|\|\\overline\{U^\{s,\\gamma\}\}\|=\|U^\{s,\\gamma\}\|\)\. We recompute all 7 yes\-rate vectors and 7 Spearman correlationsρΓc\\rho\_\{\\Gamma\_\{c\}\}\. ##### Named pair family Forc=7c=7we have eight tests: six adjacent pairs\(Λ,Ψ\)∈\(1,2\),\(2,3\),\(3,4\),\(4,5\),\(5,6\),\(6,7\)\(\\Lambda,\\Psi\)\\in\(1,2\),\(2,3\),\(3,4\),\(4,5\),\(5,6\),\(6,7\)plus the extreme pair\(Λ,Ψ\)=\(1,7\)\(\\Lambda,\\Psi\)=\(1,7\)\. We usepΛ,ΨFWERp^\{\\text\{FWER\}\}\_\{\\Lambda,\\Psi\}to control for sub\-group dependencies setting the null statistic to theλ,ψ\\lambda,\\psithat yield the maxΔλ,ψ¯\\overline\{\\Delta\_\{\\lambda,\\psi\}\}in every permutation\.

Worked Example – Multi\-Category Case: Agec=7c=7\(Cont\.\)##### Results afterz=105z=10^\{5\}permutations: ##### Range test: The observed maximum correlation isρΓ3=0\.7895\\rho\_\{\\Gamma\_\{3\}\}=0\.7895\(35–44 yr\) and the minimum isρΓ7=0\.7105\\rho\_\{\\Gamma\_\{7\}\}=0\.7105\(75\+, see Table above\), givingΔ3,7=0\.7895−0\.7105=0\.0790\\Delta\_\{3,7\}=0\.7895\-0\.7105=0\.0790\. This is larger than the pre\-specified extreme pair gapΔ1,7=0\.0545\\Delta\_\{1,7\}=0\.0545, reflecting that 35–44 year\-olds, not the youngest cohort, are most aligned with GPT\-5\.2\. The range nullΔλ,ψ¯\\overline\{\\Delta\_\{\\lambda,\\psi\}\}automatically accounts for the data\-driven selection of this pair, yieldingprange=2100,000≈2×10−5=\(\*\*\*\)\.p^\{\\mathrm\{range\}\}=\\frac\{2\}\{100,000\}\\approx 2\\times 10^\{\-5\}=\(\\text\{\*\*\*\}\)\.The age gradient in LLM alignment is highly significant, with the oldest respondents \(75\+\) being the least aligned\.

### 4\.3Topic\-Specific Attributes \(Topic\-Specific Knowledge, View Intensity\)

Topic\-specific knowledge and View intensity are reported separately for each of the three topics \(conflicts\) — Russia–Ukraine, Israel–Palestine, and Trump–Harris — because each respondent reports a separate level for each topic \(e\.g\., a respondent may be “Knows a great deal” on Russia–Ukraine while being “Heard, knows nothing” on Trump–Harris\)\. For these attributes, level membership is therefore a function of both the respondent and the topic of the question\.

Let𝒯=\{Rus–Ukr, Isr–Pal, Tru–Har\}\\mathcal\{T\}=\\\{\\text\{Rus\-\-Ukr, Isr\-\-Pal, Tru\-\-Har\}\\\}index topics, letℓt​\(u\)∈\{1,…,c\}\\ell\_\{t\}\(u\)\\in\\\{1,\\dots,c\\\}denote respondentuu’s level on topict∈𝒯t\\in\\mathcal\{T\}, and letQt⊆QQ^\{t\}\\subseteq Qbe the questions belonging to topictt, with⋃tQt=Q\\bigcup\_\{t\}Q^\{t\}=Q; and\|Qt\|=144\|Q^\{t\}\|=144\(=Q/3=Q/3\)\.

##### Per\-topic pools and observed correlations

For each topicttand levelcc, the corresponding subgroup isUt,c=\{u∈U:ℓt​\(u\)=c\}U^\{t,c\}=\\\{u\\in U:\\ell\_\{t\}\(u\)=c\\\}\. the level\-ccyes\-rate on questionq∈𝒬tq\\in\\mathcal\{Q\}^\{t\}is computed only from respondents whose level on the question’s topic equalscc, soric=yicmicr\_\{i\}^\{c\}=\\frac\{y^\{c\}\_\{i\}\}\{m^\{c\}\_\{i\}\}, wheremic=\|Ut​c\|m^\{c\}\_\{i\}=\|U^\{tc\}\|– the total number of responders with levelccon topictt, andyicy^\{c\}\_\{i\}is the number of responders inUt​cU^\{tc\}answering positively in questionqi∈Qtq\_\{i\}\\in Q^\{t\}\. Concatenating over topics, we haveRcR^\{c\}of the full dimensionkkthat could be correlated withRMR^\{M\}\.

##### Per\-topic joint shuffle

Within each topictt*independently*and within each Splitssinside that topic, we permute the level labelsℓt\\ell\_\{t\}among respondents at any of thecclevels, preserving the per\-Split per\-sub\-demographic \(/level\) counts as specified in the stratified permutation function in Equation[1](https://arxiv.org/html/2607.27232#S4.E1)\. The independence across topics reflects the fact that each respondent’s three topic\-specific levels are separate observations and may be relabeled independently under the null\.

Adjacent, extreme, and range statistics, and the correspondingpp\-valuespraw,pFWER,prangep^\{\\mathrm\{raw\}\},p^\{\\mathrm\{FWER\}\},p^\{\\mathrm\{range\}\}, are the same as defined above\.

### 4\.4Summary of Results

Tables[2](https://arxiv.org/html/2607.27232#S4.T2)and[3](https://arxiv.org/html/2607.27232#S4.T3)report statistical significance for the alignments differences between pairs of demographic subgroups and the GPT\-5\.2 model\. Significance scores are based on all questions inQQ\(432\) The number of permutations per test isz=105z=10^\{5\}\. The family of*Adjacent*and*extreme*pairs are reported throughpFWERp^\{\\mathrm\{FWER\}\}\(max\-statistic correction within Family of significance tests\);*max/min*useprangep^\{\\mathrm\{range\}\}directly \(— in thepF​W​E​Rp^\{FWER\}column\)\. Significance codes: \*\*\*p<0\.001p<0\.001, \*\*p<0\.01p<0\.01, \*p<0\.05p<0\.05, ns = not significant\.

Table 2:Significance tests for various demographic sub\-groups \(z=105z=10^\{5\}, M = GPT\-5\.2,\|Q\|=432\|Q\|=432questions\)AttributeRoleComparisonnAn\_\{A\}nBn\_\{B\}Δ​ρ\\Delta\\rhoprawp^\{\\mathrm\{raw\}\}pFWERp^\{\\mathrm\{FWER\}\}Sig\.GendersingleFemale vs Male1,7691,242−0\.0007\-0\.00070\.9020\.902nsFirst languagesingleEnglish vs Non\-English2,720291\+0\.0411\+0\.04110\.0060\.006\*\*EducationadjacentNo qual\. vs Secondary101859−0\.0830\-0\.08300\.0120\.013\*adjacentSecondary vs Undergraduate859775−0\.0092\-0\.00920\.2130\.994nsadjacentUndergraduate vs Masters/PhD775397\+0\.0057\+0\.00570\.6090\.999nsextremeNo qual\. vs Masters/PhD101397−0\.0865\-0\.08650\.0040\.008\*\*Education*min/max*Undergraduate vs No qual\.775101\+0\.0922\+0\.09220\.004—\*\*Social gradeadjacentA vs B422625−0\.0137\-0\.01370\.1920\.730nsadjacentB vs C1625917\+0\.0054\+0\.00540\.5160\.995nsadjacentC1 vs C2917449\+0\.0041\+0\.00410\.7020\.998nsadjacentC2 vs D449311\+0\.0239\+0\.02390\.0510\.179nsadjacentD vs E311287\+0\.0029\+0\.00290\.8211\.000nsextremeA vs E422287\+0\.0225\+0\.02250\.0790\.231nsSocial grade*min/max*B vs E625287\+0\.0362\+0\.03620\.034—\*Ageadjacent18–24 vs 25–34283498−0\.0201\-0\.02010\.1340\.543nsadjacent25–34 vs 35–44498586−0\.0043\-0\.00430\.6431\.000nsadjacent35–44 vs 45–54586494\+0\.0045\+0\.00450\.6270\.999nsadjacent45–54 vs 55–64494521\+0\.0215\+0\.02150\.0240\.476nsadjacent55–64 vs 65–74521438\+0\.0124\+0\.01240\.2140\.901nsadjacent65–74 vs 75\+438191\+0\.0406\+0\.04060\.0230\.033\*extreme18–24 vs 75\+283191\+0\.0545\+0\.05450\.00080\.002\*\*Age*min/max*35–44 vs 75\+586191\+0\.0790\+0\.07903×\\times10\-5—\*\*\*Table 3:Significance tests for Topic\-Awareness and View intensity levels \(z=105z=10^\{5\}, M = GPT\-5\.2,\|Q\|=432\|Q\|=432questions\)\.

## 5Alignment across Topics

![Refer to caption](https://arxiv.org/html/2607.27232v1/topic_models_heatmap.png)Figure 6:Models’ alignment score per each of the three different topicsMistral’s markedly lower alignment \(ρ=0\.41\\rho=0\.41\) is not an artifact of unparsable or refused responses \- coercion rates were negligible across all models\. Rather, it reflects two compounding tendencies\. First, Mistral answered almost deterministically, pinning 98% of its question\-level scores to 0 or 1 \(versus 11% of human responses\), which collapses the graded signal needed to track human sympathy proportions\. Second, and more decisively, its weakness was concentrated almost entirely in a single topic: while it achieved moderate alignment on the Russo\-Ukrainian war \(ρ=0\.59\\rho=0\.59\) and the Trump–Harris campaign \(ρ=0\.52\\rho=0\.52\), its alignment on the Israel–Palestine headlines was effectively random \(ρ=0\.09\\rho=0\.09, versus 0\.78 for GPT\-5\.2\)\. On these items Mistral treated Israel\-sympathy as a near coin\-flip uncorrelated with human readers and systematically under\-registered Gaza sympathy, defaulting to ”No” on 75% of headlines even though human respondents leaned sympathetic\. Thus Mistral’s poor aggregate performance masks a topic\-specific collapse rather than uniformly weak comprehension \- demonstrating how models’ alignment could differ per topic, not only demographic variables\. Because this phenomenon was largely unique to Mistral in our dataset \- at least in its magnitude \- we report the pooled results in the main text to preserve readability and interpretability, while providing topic\-specific analyses in the supplemental material \(see Figure[6](https://arxiv.org/html/2607.27232#S5.F6)\)\.