Evaluating Pluralism in LLMs through Latent Perspectives

# Evaluating Pluralism in LLMs through Latent Perspectives
Source: [https://arxiv.org/html/2606.13254](https://arxiv.org/html/2606.13254)
###### Abstract

The growing need to represent diverse perspectives has increased interest in pluralistic LLM generation\. Although difficult to operationalize, identifyingperspectivesexpressed in text would provide clear guidance on pluralistic alignment and more clearly articulate the pluralistic gap in LLM generation\. While models have been shown to reduce the diversity of training data and generate homogeneously, this has been demonstrated primarily on multiple\-choice questionnaires or using high\-level characteristics of free\-form text\. In this paper, we introduce and implement a domain\-agnostic multi\-layered framework for unsupervised extraction of perspectives suitable for identifying the pluralistic gap in LLM\-generated text\. We evaluate our framework on book reviews, a highly opinionated dataset representing diverse perspectives, and compare various prompts and models\. Our results show that while some models and prompting techniques come close to covering a broad spectrum of perspectives, rarer perspectives remain disproportionately underrepresented, resulting in distributions that diverge from human text\.

Machine Learning, ICML

## 1Introduction

As large language models scale, their competence across a broad range of tasks such as coding, math, and complex reasoning also improves\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib8); Phanet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib9)\)\. However, performance on tasks with objectively verifiable outputs does not necessarily indicate alignment to the inherent diversity of human perspectives in the myriad of subjective tasks where “correctness” is defined by specific cultural, social, or individual frameworks rather than universal facts\(Frendaet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib29); Fleisiget al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib27)\)\. To provide nuanced and balanced viewpoints on open\-ended subjective tasks, language models should undergopluralistic alignment\(Sorensenet al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib1)\), that is, they should be optimized to adequately represent the diversity of perspectives of a target population in their generated responses\. Concerningly, current empirical evidence suggests that frontier models fall short of this ideal: LMs exhibit a lower linguistic variety\(Russoet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib5)\)and a high homogeneity in their generated texts – a phenomenon dubbed thegenerative monoculture\(Wuet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib3)\)\. In general, models produce both inter\- and intra\-homogeneous outputs, generating repetitive text that is also highly similar across model families\(Jianget al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib10)\)\.

![Refer to caption](https://arxiv.org/html/2606.13254v1/x1.png)Figure 1:The proposed pluralistic evaluation framework, used for extracting perspectives and evaluating their diversity in human and LLM\-generated data\. We first identify aspects from text \(1\), cluster them \(2\), producing perspective representations \(3\), which we cluster again to identify collective perspectives \(4\)\. Across levels, we evaluate aspect level coverage \(A\), features of the perspective representation \(B\), and perspective coverage \(C\)\.One way to assess the degree of pluralism represented by a model is to analyse the diversity of*perspectives*represented in the outputs it generates\. Perspective, in the most abstract sense defined as “a particular way of considering something”\(Cambridge University Press,[n\.d\.](https://arxiv.org/html/2606.13254#bib.bib30)\), is embedded in every communication act\(Basileet al\.,[2022](https://arxiv.org/html/2606.13254#bib.bib26)\)\. In NLP, perspectives are generally used as an umbrella term for subjective language and are considered a facet of pluralistic alignment\(Sorensenet al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib1)\)\. Although perspective is difficult to operationalize, it is thelatent driverbehind overt manifestations such as sentiment, opinions, and claims, and it provides a powerful framework for evaluating model pluralism by unifying these otherwise disparate constructs into a holistic lens\.

The ability of LLMs to model human perspectives was previously evaluated primarily using MCQ surveys\(Santurkaret al\.,[2023](https://arxiv.org/html/2606.13254#bib.bib38)\)and Likert scales\(Meisteret al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib37)\)\. Where free\-form text has been considered, homogeneity has been measured using aggregated, high\-level features\(Wuet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib3)\), with little attention to the range of perspectives expressed in human text\. By reducing perspectives to aggregated labels, prior approaches fail to model the very complexity they seek to measure, ultimately masking the extent of thepluralistic gapin model generation\.

To address these limitations, we introduce a domain\-agnostic framework that formalizes*latent perspective*as a composite of aspects extracted from text, offering a fine\-grained alternative to surface\-level homogeneity metrics\. By using aspects as building blocks to characterize the collective perspectives expressed in free\-form text, our method enables a high\-resolution comparison of how perspective distributions vary between any diverse text collections, but is particularly suitable for human and LLM\-generated content\. A visualization of our framework is shown in[Figure˜1](https://arxiv.org/html/2606.13254#S1.F1)\. We test the applicability of our framework on a corpus that is diverse, opinionated, and with clear topics – a book review dataset sourced from the Goodreads platform\(Wan and McAuley,[2018](https://arxiv.org/html/2606.13254#bib.bib4)\)\. We evaluate pluralism across open\-source \(Llama 3 8B,OLMo2 1B & 8B\) and proprietary \(GPT 4\.1andGemini 2\.5\) models\. To elicit higher coverage of perspectives in LLM outputs, we use high temperature as well as persona prompting\(Geet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib15)\), a technique applied heavily in population simulation\. We apply our framework to the LLM\-generated reviews and assess their pluralism using various quantitative distributional measures, including divergence and coverage, as well as a qualitative analysis to identify topics underrepresented in LLM text\.

Our framework allows for a unified view of both the plurality of perspectives present in long\-form LLM outputs, and the comparison with human text\. Our results confirm previous findings regarding the surface\-level homogeneity of LLM outputs and also demonstrate that when evaluating pluralism in the context of perspective, models vary significantly in both baseline performance and their sensitivity to prompting\. We find that, while some models and configurations might come close to encompassing a spectrum of diverse aspects \(Overton pluralism\), rarer aspects remain disproportionately underrepresented \(distributional pluralism\), leading to a homogeneity in majority perspectives\. Our contributions can be summarized as follows:\(i\)we propose a multi\-leveled framework for comparing latent perspectives expressed in two sources of text, suitable for analysing model pluralism,\(ii\)we implement the framework on the opinionated dataset of book reviews across different LLMs and prompting methods designed to elicit diverse outputs, and\(iii\)we analyse perspective gaps at various levels of abstraction and differences across models and prompting techniques\.

## 2Related Work

The tendency of LLMs to flatten the diversity of human perspectives by converging toward a more uniform distribution of features and semantics in their outputs has been studied under several names, includingartificial hivemind\(Jianget al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib10)\),generative monoculture\(Wuet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib3)\), anddistributional gap\(Peeperkornet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib7)\), and reframed constructively as the effort for higher modelpluralism\.Sorensenet al\.\([2024](https://arxiv.org/html/2606.13254#bib.bib1)\)propose three pluralistic alignment modes:Overton, representing the set of possible answers,distributional, representing the distribution of possible answers, andsteerable, representing output similar to a chosen group to steer towards\. Evaluation of these modes in prior work is predominantly discrete: opinion alignment is operationalised through MCQ choices or Likert ratings\(Santurkaret al\.,[2023](https://arxiv.org/html/2606.13254#bib.bib38); Durmuset al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib17); Meisteret al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib37)\), or reduced to classification accuracy over attribute profiles and preference rankings\(Adamset al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib18); Chenet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib20)\)\. These approaches share a common limitation by reporting aggregate statistics over a population, discarding fine\-grained information indicating how opinions are framed and which aspects are pivotal\.

Fewer works study output homogenisation directly on the features of free\-form text, as opposed to the projected labels\.Wuet al\.\([2025](https://arxiv.org/html/2606.13254#bib.bib3)\)demonstrate generative monoculture in book reviews by measuring diversity through extracted attributes \(binary sentiment and coarse topic labels\), whilePeeperkornet al\.\([2025](https://arxiv.org/html/2606.13254#bib.bib7)\)measure the diversity gap in narrative generation via aggregate scalar scores such as the Vendi Score\. While both argue that LLM\-generated text exhibits a more narrow distribution of features compared to human text on the matching tasks and topics, their metrics collapse the distributional structure of generated text into a single diversity score, making it difficult to identify which perspectives are systematically absent\. A deeper, more structured evaluation is necessary, one that takes into account the multifaceted properties of perspective\.

The limited evaluation on free\-form text largely reflects the difficulty of working with it\. Even within perspectivism\(Frendaet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib29)\)– a paradigm that emphasizes preserving diverse perspectives across NLP tasks – perspective is typically operationalized as task\-specific labels assigned to instances\. These labels can then support various modeling and analysis approaches, such as creating annotator profiles using clustering\(Vitsakiset al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib22)\), but perspective is still effectively anchored to a label space, leaving the nuances of free\-form expression largely unaddressed\. Clustering has been used as a method of identifying common elements in various subjective tasks, both in argumentation\(Reimerset al\.,[2019](https://arxiv.org/html/2606.13254#bib.bib25)\)and framing\(Ajjouret al\.,[2019](https://arxiv.org/html/2606.13254#bib.bib23)\), though it has not been used for measuring and comparing the the homogeneity of LLMs, a gap which we fill with this paper\.

## 3A Framework for Perspective Analysis

### 3\.1Motivation

Perspectives in text are expressed through inherently subjective language, influenced by individual interpretation, emphasis, and context\. A prerequisite for automatic extraction of perspective is operationalizing it, which is difficult because precise definitions and appropriate levels of granularity are context\-sensitive\. This results in two central challenges: \(1\) how to construct a meaningful representation of diverse perspectives on a given topic and \(2\) how to compare the distribution of perspectives from different data sources, such as human and LLM\-generated text\.

To address these challenges, we propose a two\-level framework for identifying perspectives identified in a collection of texts and assessing their diversity\. The first level operates at the level ofaspects, where the representations of individual discourse units from texts are isolated and grouped to capture recurring semantic patterns\. The second level operates at theperspective level, using coarse labels of aspect groups produced at the previous level to characterize the perspective, both underlying and verbalized in the text\. Our framework is grounded in the idea of collective perspectives, identifying recurring patterns expressed across individuals that represent both majority and minority viewpoints, without imposing constraints on their form or content\.

The proposed framework enables a coarse\-grained analysis of perspectives from free\-form text, as well as a fine\-grained comparison of the underlying aspects at the first level\. A concrete implementation of the framework requires operationalizing its core components: the first level of aspect segmentation and identification of collective aspects, the second level where perspectives are constructed from aspects and grouped, and defining metrics that compare perspective diversity from different sources\. In the following section, we provide the formal definition of the framework \(§[3\.2](https://arxiv.org/html/2606.13254#S3.SS2)\), followed by a concrete implementation \(§[3\.3](https://arxiv.org/html/2606.13254#S3.SS3)\), which we then apply to a book corpus \(§[5](https://arxiv.org/html/2606.13254#S5)\) and finally rigorously verify the validity of its components \(§[6](https://arxiv.org/html/2606.13254#S6)\)\.

### 3\.2Definition

LetD=\{d1,d2,…,dn\}D=\\\{d\_\{1\},d\_\{2\},\\ldots,d\_\{n\}\\\}be a dataset of texts sharing a common topic, such as reviews of the same book\.

##### Aspects\.

Each textdid\_\{i\}can be decomposed into a sequence ofmim\_\{i\}aspect instances:

di=\[a1i,a2i,…,amii\]d\_\{i\}=\[a^\{i\}\_\{1\},a^\{i\}\_\{2\},\\ldots,a^\{i\}\_\{m\_\{i\}\}\]\(1\)where eachajia^\{i\}\_\{j\}represents the aspect expressed in unitjjof textii\. Common aspects should then be grouped acrossDD, yielding a set ofkkaspect clusters𝒞=\{C1,…,Ck\}\\mathcal\{C\}=\\\{C\_\{1\},\\ldots,C\_\{k\}\\\}\. Each aspect instanceajia^\{i\}\_\{j\}is assigned to exactly one clusterCl∈𝒞C\_\{l\}\\in\\mathcal\{C\}, or designated as an outlier and excluded:

aji↦\{Clif​aji​belongs to a recognised cluster∅if​aji​is an outliera^\{i\}\_\{j\}\\mapsto\\begin\{cases\}C\_\{l\}&\\text\{if \}a^\{i\}\_\{j\}\\text\{ belongs to a recognised cluster\}\\\\ \\varnothing&\\text\{if \}a^\{i\}\_\{j\}\\text\{ is an outlier\}\\end\{cases\}\(2\)

##### Perspective\.

We define the perspective of a textdid\_\{i\}as the distribution over clusters induced by its non\-outlier aspect instances\. Letnlin^\{i\}\_\{l\}denote the number of aspect instances indid\_\{i\}assigned to clusterClC\_\{l\}, and letNi=∑l=1knliN\_\{i\}=\\sum\_\{l=1\}^\{k\}n^\{i\}\_\{l\}be the total number of non\-outlier aspect instances\. We define the perspective ofdid\_\{i\}as the vector:

𝐩i=\(n1iNi,n2iNi,…,nkiNi\)∈Δk−1\\mathbf\{p\}\_\{i\}=\\left\(\\frac\{n^\{i\}\_\{1\}\}\{N\_\{i\}\},\\ \\frac\{n^\{i\}\_\{2\}\}\{N\_\{i\}\},\\ \\ldots,\\ \\frac\{n^\{i\}\_\{k\}\}\{N\_\{i\}\}\\right\)\\in\\Delta^\{k\-1\}\(3\)whereΔk−1\\Delta^\{k\-1\}denotes the\(k−1\)\(k\-1\)\-dimensional probability simplex\.

### 3\.3An Implementation of the Framework

#### 3\.3\.1Aspect level

In our work, we choose a sentence as the target discourse unit representing an aspect, and use sentence embeddings as its semantic representation within the framework\. We do not construct a structured representation of aspects, which keeps the approach context\-independent\. We provide a more structured analysis \(§[6\.1](https://arxiv.org/html/2606.13254#S6.SS1)\) and validate the quality of this representation \(§[6\.2](https://arxiv.org/html/2606.13254#S6.SS2)\) in later sections\. To estimate the distribution of aspects present in book reviews, we \(1\) segment the original reviews into sentences, then \(2\) embed each sentence using a sentence encoder model, and finally \(3\) group the embeddings into aspect clusters\.

##### Implementation\.

We use spaCy to segment free\-form text into sentences\.111[https://spacy\.io/models/en](https://spacy.io/models/en)We opt for the F2LLM 0\.6B model\(Zhanget al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib13)\)as the sentence embedder, as it currently achieves the best clustering performance among sub\-1B parameter English models on the MTEB leaderboard\.222[https://huggingface\.co/spaces/mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard)To cluster the aspects encoded within sentences, we build on the BERTopic\(Grootendorst,[2022](https://arxiv.org/html/2606.13254#bib.bib14)\)library, and choose HDBSCAN\(Campelloet al\.,[2013](https://arxiv.org/html/2606.13254#bib.bib16)\)as the aspect\-level clustering algorithm, as unlike centroid\-based methods such as k\-means, it does not require the number of clusters to be specified a priori\. This property is crucial in our setting, as the number of collective aspect clusters in a group of reviews is not known beforehand, and predefining it would impose unnecessary constraints\. Prior to clustering, we reduce the dimensionality of sentence embeddings to55using UMAP\(McInneset al\.,[2018](https://arxiv.org/html/2606.13254#bib.bib36)\)\.333We ran experiments covering higher dimensionality with comparable or worse results\.

We select 1500 human reviews per book for our analysis, and split them into a fixed base \(R𝑏𝑎𝑠𝑒R\_\{\\mathit\{base\}\}\) and evaluation \(Re​v​a​lR\_\{eval\}\) set, with\|R𝑏𝑎𝑠𝑒\|=1000\|R\_\{\\mathit\{base\}\}\|=1000and\|R𝑒𝑣𝑎𝑙\|=500\|R\_\{\\mathit\{eval\}\}\|=500\. We compute the clusters on the base set, and use the held\-out evaluation set for comparison with the same number of LLM\-generated reviews\. We opt for this split to ensure a larger set of reviews that serves both as a stable base for cluster computation and as a more accurate approximation of the full aspect distribution, while the smaller number of held\-out reviews allows for an apples\-to\-apples comparison with LLM\-generated reviews\. The sentencized and embedded reviews fromR𝑏𝑎𝑠𝑒R\_\{\\mathit\{base\}\}are used to fit a clustering model for each book \(bb\), resulting inKbK\_\{b\}clusters per book\. Sentences fromR𝑒𝑣𝑎𝑙R\_\{\\mathit\{eval\}\}and LLM\-generated reviews are then assigned aspect cluster labels using the fit clustering models\.

##### Evaluation\.

We use two quantitative metrics to evaluate aspect\-level differences between human and LLM\-generated texts\. If LLMs are properly pluralistically aligned, the cluster assignment compared to human texts should be \(1\) complete, covering all clusters to reflect*Overton pluralism*, and \(2\) proportionate, maintaining the same ratios to reflect*distributional pluralism*\. We operationalize these using cluster coverage percentage and Jensen\-Shannon Divergence \(JSD\), respectively, measured between the cluster distributions at the book level, then averaged across all books\.

Apart from these contrastive measures, we also compare individual population statistics, which measure aspect diversity within reviews originating from a single source\. As these statistics, we opt for average semantic similarity, measuring sentence homogeneity, and aspect cluster entropy\.

#### 3\.3\.2Perspective level

To aggregate the individual aspects to the perspective level, we encode each review as a set of aspects based on the cluster assignments of its constituent sentences\. Since HDBSCAN is a probabilistic model, its output can be interpreted either as a distribution overKbK\_\{b\}aspect clusters or as the most probable cluster\. We opt to use the most probable cluster as the label, constructing the perspective representation as the count of aspects in that cluster\. The individual representations are then clustered to recognise collective perspectives\.

##### Implementation\.

After constructing the perspective vectors in the same human \(R𝑏𝑎𝑠𝑒R\_\{\\mathit\{base\}\},R𝑒𝑣𝑎𝑙R\_\{\\mathit\{eval\}\}\) and LLM\-generated sets, we implement another level of clustering models to determine the collective perspectives\. In contrast to the aspect level, perspective vectors are sparser, making the previous density\-based HDBSCAN model no longer effective\. Therefore, we utilize other clustering models, including k\-means and community detection\.

##### Evaluation\.

Akin to the aspect level, we evaluate the distribution of perspectives using total cluster coverage and the Jensen\-Shannon Divergence \(JSD\) to explicitly compare Overton and distributional pluralism\. We complement this by cosine similarity of the perspective representation vectors to assess population diversity\.

## 4Dataset and Models

### 4\.1Dataset

We utilize the Goodreads dataset\(Wan and McAuley,[2018](https://arxiv.org/html/2606.13254#bib.bib4)\)for all experiments\. The dataset consists of book reviews and corresponding metadata across various genres\. We chose this dataset because it contains opinionated texts not explicitly related to political stances\. Furthermore, the 2017 data cut\-off nullifies the risk of LLM\-generated text\.

For our analysis, we select from English\-language books that contain more than 1500 reviews\. To ensure a representative sample, we sample 20 books from each genre, including five from each of the following categories: highest average score, lowest average score, highest score deviation, and highest number of reviews\. The represented genres areMystery, Thriller and Crime,Young Adult,History and Biography,Fantasy and Paranormal, andRomance\.

From these, we select a preliminaryanalysis subsetof 10 books, sampled across genres, for initial testing and analyses\. We filter out reviews not written in English or those shorter than 20 characters, usinglangdetectfor language detection\.444[https://pypi\.org/project/langdetect/](https://pypi.org/project/langdetect/)

### 4\.2Models and prompting configurations

We run our experiments across various closed\- and open\-source LLMs\. For closed\-source models, we selectgpt\-4\.1\-mini\-2025\-04\-14\(Achiamet al\.,[2023](https://arxiv.org/html/2606.13254#bib.bib31)\)andgemini\-flash\-2\.5\-mini\(Comaniciet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib33)\)\(hereafter abbreviated asGPT 4\.1andGemini 2\.5\)\. For open\-source models, we selectLlama\-3\.1\-8B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib21)\)as well asOLMo\-2\-1BandOLMo\-2\-8B\(OLMoet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib12)\)\. The availability of pre\-training corpora for OLMo models allows us to verify whether they were exposed to the review texts\.

We measure and compare the diversity of perspectives in book reviews generated by humans and LLMs under different prompting setups\. We generate 300 reviews for the 10\-book subset and 100 reviews for the full 100\-book set\. We utilize three different prompting setups:

##### Baseline\.

Baselineprompts contain the vanilla instruction to write a book review, as well as rate it\. We use a temperatureT=0\.7T=0\.7to obtain variety across samples\.

##### High temperature\.

Higher temperature values are often used in creative tasks\(Wuet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib3)\)\. To evaluate the influence of higher temperature on review diversity, we use the baseline prompt and evaluate it atT=1\.5T=1\.5\.

##### Persona prompts\.

Socio\-demographic prompting and persona prompting are promising ways to increase the diversity of LLM output\. We select a fixed batch of300300personas fromGeet al\.\([2025](https://arxiv.org/html/2606.13254#bib.bib15)\)and use each persona to generate a single review\.

We report prompts used across all setups in Appendix[A](https://arxiv.org/html/2606.13254#A1)\.

## 5Results

In this section, we first present aggregate measures \(§[5\.1](https://arxiv.org/html/2606.13254#S5.SS1)\) and then fine\-grained aspect\- and perspective\-level comparisons from our framework \(§[5\.2](https://arxiv.org/html/2606.13254#S5.SS2)\)\.

### 5\.1High\-level measures

We first perform a set of analyses focusing on high\-level, aggregated measures to assess consistency with prior findings and to confirm differences in superficial features between the baseline and other setups\. We report results on the analysis subset of 10 books, comparing 300 model\-generated reviews with 300 held\-out human reviews fromR𝑒𝑣𝑎𝑙R\_\{\\mathit\{eval\}\}\.

##### Semantic similarity\.

To measure the novelty of generated reviews across runs, we process reviews for each book in sequential batches of 10\. For each new batch, we compute the cosine similarity between each sentence embedding in the incoming batch and all sentence embeddings from previous batches, recording the maximum value\. We then average this score across books to obtain a mean\-max similarity score\. A high mean\-max similarity indicates that new reviews are progressively more repetitive, while a low value suggests they introduce novel content\.

Evaluating the original and generated reviews separately allows us to compare how quickly each source saturates\.[Figure˜2](https://arxiv.org/html/2606.13254#S5.F2)shows the mean\-max similarity curves for human reviews and those generated byGPT 4\.1\. The results point to two major findings: \(1\) diversity saturates at around 100 reviews, leading us to opt for that sample size in further experiments, and \(2\) the semantic similarity of human reviews is lower than that of generated reviews, withpersonaprompting having a positive influence on review diversity\. We report the results for other models in Appendix[B](https://arxiv.org/html/2606.13254#A2)\.

![Refer to caption](https://arxiv.org/html/2606.13254v1/x2.png)Figure 2:Mean maximum cosine similarity between embeddings for original reviews and across generation configurations forGPT 4\.1\. Results for other models are presented in Appendix[B](https://arxiv.org/html/2606.13254#A2)\.
##### Sentiment and rating\.

The review dataset is highly subjective, where attitudes toward a book are conveyed both implicitly through sentiment and explicitly through assigned ratings ranging from 1 to 5\.Wuet al\.\([2025](https://arxiv.org/html/2606.13254#bib.bib3)\)show LLM\-generated reviews are more positive in sentiment than original reviews, but do not compare the assigned rating\. We estimate sentiment of book reviews usingdistilbert\-base\-uncased\-finetuned\-sst\-2\-english\(Sanhet al\.,[2019](https://arxiv.org/html/2606.13254#bib.bib32)\)\. The sentiment range is\[−1,1\]\[\-1,1\], with−1\-1denoting negative and\+1\+1positive sentiment, respectively\.

We report the average sentiment and assigned ratings across books for original and generated reviews in[Table˜1](https://arxiv.org/html/2606.13254#S5.T1)\. Results differ across models, withGemini 2\.5producing the lowest average rating and sentiment, andGPT 4\.1the highest\. Similar to our findings for semantic similarity, the baseline prompts are farthest from the original scores, scoring highly on both average sentiment \(a trend most pronounced forGPT 4\.1\) and average assigned rating\. Higher\-temperature sampling \(T=1\.5T=1\.5\) yields mixed results, typically reducing the average rating and sentiment while increasing the corresponding standard deviation\. Persona prompting generates ratings closest to the original scores; notably,Gemini 2\.5produces lower average ratings than the original reviews\.

Table 1:Mean±std\{\}\_\{\\pm\\text\{std\}\}book review rating and detected sentiment per model and generation configuration\.

### 5\.2Framework results

##### Aspect level\.

To compare the topic distributions of human and LLM\-generated reviews, we use two measures\. First, we estimate alignment with Overton pluralism, reflecting the breadth of the generating corpus throughcluster coverage, which measures the percentage of human\-identified clusters that appear at least once in the generated reviews\. Second, to estimate distributional pluralism – whether LLM\-generated aspects are discussed in proportions similar to those of human reviewers – we compute the JSD, quantifying how much the generated corpus diverges from the human baseline in aspect distributions\.

We report results across models and configurations in[Table˜2](https://arxiv.org/html/2606.13254#S5.T2)\. Persona prompting consistently yields higher topic coverage and lower divergence from the human distribution than temperature\-based sampling across all models and metrics\.Gemini\-2\.5under persona prompting reaches the highest coverage \(98\.0±2\.6%98\.0\\pm 2\.6\\%\) and lowest divergence \(JSD=0\.11±0\.02\\text\{JSD\}=0\.11\\pm 0\.02\), suggesting it most closely approximates both Overton and distributional pluralism\. Interestingly,GPT\-4lags behind bothGemini\-2\.5and the smallerLlama\-3\.1\-8Bacross all configurations, peaking at only62\.7±3\.4%62\.7\\pm 3\.4\\%coverage under persona prompting\. A higher temperature \(T=1\.5T=1\.5vs\.T=0\.7T=0\.7\) yields modest but consistent gains in both metrics\. Overall, we find that the choice of prompting strategy has a stronger effect on topical diversity than temperature scaling and that persona prompting can effectively narrow the pluralistic gap\.

Table 2:Cluster coverage and JSD \(mean±std\{\}\_\{\\pm\\text\{std\}\}across books\)\. Cluster coverage estimates Overton pluralism, while JSD estimates distributional pluralism\.Table 3:Intra\-cluster semantic similarity and entropy \(mean±std\{\}\_\{\\pm\\text\{std\}\}across books\)\.
##### Perspective level\.

To estimate perspective\-level coverage, we use metrics similar to the aspect\-level ones: JSD, cluster coverage, and normalized entropy\. We add a novel perspective\-level metric –*perspective diversity*– which measures the average number of perspective clusters covered in a set of reviews relative to those covered in the evaluation set\. We report results of all metrics in[Table˜4](https://arxiv.org/html/2606.13254#S5.T4)for using the k\-means clustering algorithm withk=5k=5\. We find that perspective\-level coverage is considerably lower across all models than aspect\-level coverage, indicating that while constituent aspects are adequately represented in generated texts, generated perspectives remain largely homogeneous\. This homogeneity is best seen through the perspective diversity metric, which is considerably lower than in the human reviews fromR𝑒𝑣𝑎𝑙R\_\{\\mathit\{eval\}\}covering nearly all perspective clusters, with LLM\-generated reviews frequently mapping to only one or two perspective clusters\. These results show that while LLMs are capable of generating texts corresponding to diverse aspects, this diversity is merely performative, with the overall perspectives present in the reviews still largelymonocultural\.

Table 4:JSD2, normalised entropy, perspective diversity, and perspective coverage \(mean±std\{\}\_\{\\pm\\text\{std\}\}across books\)\.

## 6Analysis

We now analyse the validity of our framework by evaluating cluster contents and quality and approximating the proportion of reviews present in model pre\-training sets\. We first categorize the aspect clusters into meaningful topic categories and analyse their coverage \(§[6\.1](https://arxiv.org/html/2606.13254#S6.SS1)\), then evaluate the coherence and separability of the aspect clusters \(§[6\.2](https://arxiv.org/html/2606.13254#S6.SS2)\), and finally analyse the pre\-training corpus of theOLMomodel family \(§[6\.3](https://arxiv.org/html/2606.13254#S6.SS3)\)\.

### 6\.1Topic evaluation

Results of our experiments \(§[5](https://arxiv.org/html/2606.13254#S5)\) have identified the pluralistic gap, where the LLM\-generated reviews lack diversity of perspectives\. We now aim to identify which aspects and perspectives are consistently underrepresented\. To do this, we label each inferred aspect cluster with a category and corresponding sentiment\. For each of the aspect clusters across the analysis subset, we use five sentences closest to each aspect centroid to label clusters usingGPT\-4\.1\-mini\. For labels, we use a predefined taxonomy of 24 categories proposed byYang and Jin \([2025](https://arxiv.org/html/2606.13254#bib.bib34)\)as a fixed set, making labels consistent and directly comparable across books\. The categories span both more objective \(plot, characters, writing style\) and subjective categories \(emotional impact, enjoyment, and expectation fulfillment\)\. We also assign a sentiment label \(positive, negative, neutral, or mixed\) expressed in the aspect toward that category\. The resulting labels allow us to pinpoint the reasons LLMs diverge from human reviews in the topic space, and exactly along which aspects\.

![Refer to caption](https://arxiv.org/html/2606.13254v1/x3.png)\(a\)Objective vs\. subjective coverage\.
![Refer to caption](https://arxiv.org/html/2606.13254v1/x4.png)\(b\)Parity of coverage across models\.

Figure 3:Topic coverage across generation modes for objective and subjective categories \([3\(a\)](https://arxiv.org/html/2606.13254#S6.F3.sf1)\), and parity of aspect coverage compared to original distribution \([3\(b\)](https://arxiv.org/html/2606.13254#S6.F3.sf2)\)\.To jointly evaluate the coverage of aspects across books and study the influence of prompting configurations, we aggregate aspects into subjective and objective categories, then measure the percentage of those aspects covered across configurations and models\. In[Figure˜3\(a\)](https://arxiv.org/html/2606.13254#S6.F3.sf1), we report the difference in coverage between objective \(full bar\) and subjective \(shaded bar\) aspects\. We observe that across models, persona prompting generally affects the coverage of subjective aspects, even when it does not achieve maximum coverage \(as withGPT 4\.1\)\. In OLMo models, higher temperature has a stronger influence on topic coverage than persona prompting\. Finally, model size does not necessarily imply lower coverage, asOLMo2 1Bgenerally achieves higher coverage thanGPT 4\.1\.

To complement Overton pluralism measured by topic coverage \(1 review per topic at least\), we evaluate on distributional pluralism, which is more informative\. In[Figure˜3\(b\)](https://arxiv.org/html/2606.13254#S6.F3.sf2), we compare whether the distributions of the LLM\-generated topics correspond to human ones\. These results indicate that Overton pluralism does not guarantee distributional pluralism, showing models report more spurious rather than systematic coverage of diverse topics\.

### 6\.2Cluster coherence

We now aim to verify the validity of the produced cluster assignments\. We evaluate aspect\-cluster coherence using a leave\-one\-out intrinsic probe: for each cluster, we sample four representative sentences and one intruder sentence from a different cluster\. Then, we useGPT 4\.1for LLM\-as\-a\-judge to determine the intruder\. We vary this setup across two dimensions: \(1\) whether representative sentences are drawn from the centroid or sampled randomly, and \(2\) whether the outlier comes from a random or the closest neighbouring cluster\. The LLM\-as\-a\-judge prompt is provided in Appendix[A](https://arxiv.org/html/2606.13254#A1)\.

We report the results in[Table˜5](https://arxiv.org/html/2606.13254#S6.T5)\. Intruder detection accuracy ranges from72\.4%72\.4\\%in the hardest configuration \(random–closest\) to94\.6%94\.6\\%in the easiest \(centroid–random\), against a20%20\\%random baseline, confirming that the inferred topic structure is coherent and separable\.

Table 5:Leave\-one\-out topic coherence accuracy across representative document and outlier selection strategies \(baseline = 20\.0%\)\.For the qualitative analysis, we manually reviewed the errors across all setups\. In[Table˜6](https://arxiv.org/html/2606.13254#S6.T6)we show one correct example alongside two error cases that illustrate different failure modes: one case where the mistake originated from the LLM’s identification, and another case where an example was incorrectly flagged as an outlier due to a clustering mismatch, as evidenced by its sentiment being opposite to that of the other examples in the cluster, while focused around the same topic\.

Table 6:Leave\-one\-out detection examples for identifying cluster coherence, “Pick” shows the assigned, and “Outlier” the true outlier\. We present a positive example of outlier identification, as well as two failure modes due to classification LLM error and clustering error due to a mismatch in sentiment of cluster constituents\.
### 6\.3Memorization

In this section, we aim to estimate the proportion of review data present in model pre\-training data, in order to confirm LLMs were exposed to a review pool large enough for diversity\. We conduct two complementary analyses of theDCLMcorpus\(Liet al\.,[2024](https://arxiv.org/html/2606.13254#bib.bib24)\), which is used in the pre\-training pipeline ofOLMomodels\(OLMoet al\.,[2025](https://arxiv.org/html/2606.13254#bib.bib12)\), and likely contributes to the training mixtures of other LLMs as well\. In both analyses, we first retrieve all URLs corresponding to Goodreads from DCLM, as well as alternative book\-review sources determined from manual search and LLM suggestions, indicating the presence of the sources in the training set\. The sources contain user\-generated literary reviews, book overviews, or related discussion content\. The first analysis focuses specifically on Goodreads\-derived content within DCLM, while the second examines alternative book\-review and reading\-community websites represented in the crawl\.

##### Goodreads\.

The Goodreads subset of the DCLM crawl contains25,83125,831pages from two page types:19,05219,052individual review pages \(goodreads\.com/review/show/\) and6,7796,779book\-summary pages \(goodreads\.com/book/show/\)\. Because eachreview/showpage contains one complete review, these pages contribute19,05219,052reviews directly\. Thebook/showpages additionally embed up to3030community reviews each; using truncation markers as a conservative estimate yields approximately131,441131,441further review texts, for a total estimate of150,493150,493Goodreads reviews\. Linking crawl pages to thegoodreads/booksmetadata via Goodreads IDs or normalized titles matched14,87314,873distinct books containing16\.116\.1million Goodreads reviews in total in the dataset\. The crawl captured approximately131,971131,971of these reviews, corresponding to an overall coverage rate of0\.82%0\.82\\%\. The coverage is highly skewed:32\.6%32\.6\\%of matched books have more than5%5\\%of their Goodreads reviews present in the crawl, while12\.5%12\.5\\%have coverage below0\.1%0\.1\\%\. Among the100100selected books used for downstream analysis,7777appear in the crawl, yielding612612matchedreview/showpages\. The high number of Goodreads reviews present in the training data confirms the LLMs were exposed to community reviews, validating they could feasibly reproduce the diverse perspectives present in those reviews, if properly aligned\.

##### Other sources\.

To complement the Goodreads\-focused analysis, we additionally examine review and discussion content originating from alternative reading communities and book\-review platforms present in the DCLM crawl\. We use LLM\-as\-a\-judge on a random sample of content from100100URLs of chosen book\-related domains, estimating whether and how many book reviews and book overviews were present in the page\. We then multiply the average number of reviews and overviews by the total number of pages for that domain present in the crawl\. We report complete results in Appendix[C](https://arxiv.org/html/2606.13254#A3)\. In total, the estimate comes to over300​k300kreviews and over340​k340koverviews, confirming that LLMs were exposed to a wide amount of evaluative book content\.

## 7Conclusion

In our work, we present a framework for evaluating pluralism in LLMs by first identifying constituent aspects of perspectives from free\-form text and then grouping them into perspective representations\. This two\-level scheme extends beyond previously used aggregate statistics and allows for a fine\-grained analysis of thepluralistic gapbetween human and LLM generations\. We propose one concrete instance of our framework and apply it to a dataset of book reviews, extracting aspect and perspective distributions and allowing for comparison between authentic human reviews and LLM\-generated ones\. We verify results from previous works, which indicate that LLMs are part of agenerative monoculture, but also show that the pluralistic gap can be bridged by utilizing prompting techniques such as persona prompting\. Our subsequent analyses verify the validity of our method through cluster coherence and separability and identify the coverage of which topics was improved\. Taken together, we offer a principled way of analysing fine\-grained diversity of perspectives in free\-form texts, which can be used to identify concrete targets for pluralistic alignment\.

## Impact Statement

In this work, we study pluralism in LLMs through fine\-grained analysis of aspects that are constituent of human perspectives\. We identify thepluralistic gapand evaluate whether it can be bridged using simple prompting techniques\. As LLM\-generated content becomes more prevalent and humans interact with LLMs to a greater degree, it is paramount to fairly represent individual voices, no matter how infrequent they may be\. Concerningly, various works show that LLMs sharpen the distribution of perspectives in the training data, either not encoding or not generating less frequent ones\. Our work provides a fine\-grained, structured framework for analysing this phenomenon and paves the way towards effective mitigation strategies aimed at mitigating the pluralistic gap\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.p1.1)\.
- J\. Adams, B\. Hu, E\. Veenhuis, D\. Joy, B\. Ravichandran, A\. Bray, A\. Hoogs, and A\. Basharat \(2025\)Steerable pluralism: pluralistic alignment via few\-shot comparative regression\.External Links:2508\.08509,[Link](https://arxiv.org/abs/2508.08509)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- Y\. Ajjour, M\. Alshomary, H\. Wachsmuth, and B\. Stein \(2019\)Modeling frames in argumentation\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 2922–2932\.External Links:[Link](https://aclanthology.org/D19-1290/),[Document](https://dx.doi.org/10.18653/v1/D19-1290)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p3.1)\.
- V\. Basile, T\. Caselli, A\. Balahur, and L\. Ku \(2022\)Editorial: bias, subjectivity and perspectives in natural language processing\.Frontiers in Artificial IntelligenceVolume 5 \- 2022\.External Links:[Link](https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2022.926435),[Document](https://dx.doi.org/10.3389/frai.2022.926435),ISSN 2624\-8212Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p2.1)\.
- Cambridge University Press \(n\.d\.\)Perspective\.Note:[https://dictionary\.cambridge\.org/dictionary/english/perspective](https://dictionary.cambridge.org/dictionary/english/perspective)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p2.1)\.
- R\. J\. G\. B\. Campello, D\. Moulavi, and J\. Sander \(2013\)Density\-based clustering based on hierarchical density estimates\.InAdvances in Knowledge Discovery and Data Mining,J\. Pei, V\. S\. Tseng, L\. Cao, H\. Motoda, and G\. Xu \(Eds\.\),Berlin, Heidelberg,pp\. 160–172\.External Links:ISBN 978\-3\-642\-37456\-2Cited by:[§3\.3\.1](https://arxiv.org/html/2606.13254#S3.SS3.SSS1.Px1.p1.1)\.
- Q\. Z\. Chen, K\. Feng, C\. Y\. Park, and A\. X\. Zhang \(2025\)SPICA: retrieving scenarios for pluralistic in\-context alignment\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 748–765\.External Links:[Link](https://aclanthology.org/2025.findings-acl.41/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.41),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.p1.1)\.
- E\. Durmus, K\. Nguyen, T\. Liao, N\. Schiefer, A\. Askell, A\. Bakhtin, C\. Chen, Z\. Hatfield\-Dodds, D\. Hernandez, N\. Joseph, L\. Lovitt, S\. McCandlish, O\. Sikder, A\. Tamkin, J\. Thamkul, J\. Kaplan, J\. Clark, and D\. Ganguli \(2024\)Towards measuring the representation of subjective global opinions in language models\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=zl16jLb91v)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- E\. Fleisig, S\. L\. Blodgett, D\. Klein, and Z\. Talat \(2024\)The perspectivist paradigm shift: assumptions and challenges of capturing human labels\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2279–2292\.External Links:[Link](https://aclanthology.org/2024.naacl-long.126/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.126)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1)\.
- S\. Frenda, G\. Abercrombie, V\. Basile,et al\.\(2025\)Perspectivist approaches to natural language processing: a survey\.Language Resources and Evaluation59,pp\. 1719–1746\.External Links:[Document](https://dx.doi.org/10.1007/s10579-024-09766-4)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1),[§2](https://arxiv.org/html/2606.13254#S2.p3.1)\.
- T\. Ge, X\. Chan, X\. Wang, D\. Yu, H\. Mi, and D\. Yu \(2025\)Scaling synthetic data creation with 1,000,000,000 personas\.External Links:2406\.20094,[Link](https://arxiv.org/abs/2406.20094)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p4.1),[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.SSS0.Px3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.p1.1)\.
- M\. Grootendorst \(2022\)BERTopic: neural topic modeling with a class\-based tf\-idf procedure\.arXiv preprint arXiv:2203\.05794\.Cited by:[§3\.3\.1](https://arxiv.org/html/2606.13254#S3.SS3.SSS1.Px1.p1.1)\.
- L\. Jiang, Y\. Chai, M\. Li, M\. Liu, R\. Fok, N\. Dziri, Y\. Tsvetkov, M\. Sap, and Y\. Choi \(2025\)Artificial hivemind: the open\-ended homogeneity of language models \(and beyond\)\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=saDOrrnNTz)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1),[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world Github issues?\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak,et al\.\(2024\)DataComp\-lm: in search of the next generation of training sets for language models\.External Links:2406\.11794Cited by:[§6\.3](https://arxiv.org/html/2606.13254#S6.SS3.p1.1)\.
- L\. McInnes, J\. Healy, N\. Saul, and L\. Großberger \(2018\)UMAP: uniform manifold approximation and projection\.Journal of Open Source Software3\(29\),pp\. 861\.External Links:[Document](https://dx.doi.org/10.21105/joss.00861),[Link](https://doi.org/10.21105/joss.00861)Cited by:[§3\.3\.1](https://arxiv.org/html/2606.13254#S3.SS3.SSS1.Px1.p1.1)\.
- N\. Meister, C\. Guestrin, and T\. Hashimoto \(2025\)Benchmarking distributional alignment of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 24–49\.External Links:[Link](https://aclanthology.org/2025.naacl-long.2/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.2),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p3.1),[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- T\. OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, …, and H\. Hajishirzi \(2025\)2 olmo 2 furious\.External Links:2501\.00656,[Link](https://arxiv.org/abs/2501.00656)Cited by:[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.p1.1),[§6\.3](https://arxiv.org/html/2606.13254#S6.SS3.p1.1)\.
- M\. Peeperkorn, T\. Kouwenhoven, D\. Brown, and A\. Jordanous \(2025\)Mind the gap: conformative decoding to improve output diversity of instruction\-tuned large language models\.External Links:2507\.20956,[Link](https://arxiv.org/abs/2507.20956)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p1.1),[§2](https://arxiv.org/html/2606.13254#S2.p2.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi, M\. Choi, A\. Agrawal, A\. Chopra, A\. Khoja, R\. Kim, R\. Ren, J\. Hausenloy, O\. Zhang, M\. Mazeika, S\. Yue, A\. Wang, and D\. Hendrycks \(2025\)Humanity’s last exam\.CoRRabs/2501\.14249\.External Links:[Link](https://doi.org/10.48550/arXiv.2501.14249),[Document](https://dx.doi.org/10.48550/ARXIV.2501.14249),2501\.14249Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1)\.
- N\. Reimers, B\. Schiller, T\. Beck, J\. Daxenberger, C\. Stab, and I\. Gurevych \(2019\)Classification and clustering of arguments with contextualized word embeddings\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 567–578\.External Links:[Link](https://aclanthology.org/P19-1054/),[Document](https://dx.doi.org/10.18653/v1/P19-1054)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p3.1)\.
- G\. Russo, D\. Nozza, P\. Röttger, and D\. Hovy \(2025\)The pluralistic moral gap: understanding judgment and value differences between humans and large language models\.External Links:2507\.17216,[Link](https://arxiv.org/abs/2507.17216)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§5\.1](https://arxiv.org/html/2606.13254#S5.SS1.SSS0.Px2.p1.3)\.
- S\. Santurkar, E\. Durmus, F\. Ladhak, C\. Lee, P\. Liang, and T\. Hashimoto \(2023\)Whose opinions do language models reflect?\.InProceedings of the 40th International Conference on Machine Learning,pp\. 29971–30004\.Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p3.1),[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- T\. Sorensen, J\. Moore, J\. Fisher, M\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri, T\. Althoff, and Y\. Choi \(2024\)Position: a roadmap to pluralistic alignment\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1),[§1](https://arxiv.org/html/2606.13254#S1.p2.1),[§2](https://arxiv.org/html/2606.13254#S2.p1.1)\.
- N\. Vitsakis, A\. Parekh, and I\. Konstas \(2024\)Voices in a crowd: searching for clusters of unique perspectives\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 12517–12539\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.696/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.696)Cited by:[§2](https://arxiv.org/html/2606.13254#S2.p3.1)\.
- M\. Wan and J\. J\. McAuley \(2018\)Item recommendation on monotonic behavior chains\.InProceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2\-7, 2018,S\. Pera, M\. D\. Ekstrand, X\. Amatriain, and J\. O’Donovan \(Eds\.\),pp\. 86–94\.External Links:[Link](https://doi.org/10.1145/3240323.3240369),[Document](https://dx.doi.org/10.1145/3240323.3240369)Cited by:[Table 10](https://arxiv.org/html/2606.13254#A4.T10),[Table 10](https://arxiv.org/html/2606.13254#A4.T10.3.2),[Table 9](https://arxiv.org/html/2606.13254#A4.T9),[Table 9](https://arxiv.org/html/2606.13254#A4.T9.3.2),[§1](https://arxiv.org/html/2606.13254#S1.p4.1),[§4\.1](https://arxiv.org/html/2606.13254#S4.SS1.p1.1)\.
- F\. Wu, E\. Black, and V\. Chandrasekaran \(2025\)Generative monoculture in large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yZ7sn9pyqb)Cited by:[§1](https://arxiv.org/html/2606.13254#S1.p1.1),[§1](https://arxiv.org/html/2606.13254#S1.p3.1),[§2](https://arxiv.org/html/2606.13254#S2.p1.1),[§2](https://arxiv.org/html/2606.13254#S2.p2.1),[§4\.2](https://arxiv.org/html/2606.13254#S4.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2606.13254#S5.SS1.SSS0.Px2.p1.3)\.
- D\. Yang and Q\. Jin \(2025\)What matters in evaluating book\-length stories? a systematic study of long story evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16375–16398\.External Links:[Link](https://aclanthology.org/2025.acl-long.799/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.799),ISBN 979\-8\-89176\-251\-0Cited by:[§6\.1](https://arxiv.org/html/2606.13254#S6.SS1.p1.1)\.
- Z\. Zhang, Z\. Liao, H\. Yu, P\. Di, and R\. Wang \(2025\)F2LLM technical report: matching sota embedding performance with 6 million open\-source data\.External Links:2510\.02294,[Link](https://arxiv.org/abs/2510.02294)Cited by:[§3\.3\.1](https://arxiv.org/html/2606.13254#S3.SS3.SSS1.Px1.p1.1)\.

## Appendix APrompts

We report all prompts used to generate reviews across various setups and LLM\-as\-a\-judge for cluster coherence, labelling and pretraining data analysis in[Table˜7](https://arxiv.org/html/2606.13254#A1.T7)\.

Table 7:Prompts used for generating reviews \(top\), leave\-one\-out cluster coherence evaluation \(second\), cluster labelling \(third\), and web page classification \(bottom\)\.
## Appendix BSemantic Similarity

We report the mean max semantic similarity of novel batches of reviews for other models used throughout our experiments \([Figure˜2](https://arxiv.org/html/2606.13254#S5.F2)\) in[Figure˜4](https://arxiv.org/html/2606.13254#A2.F4)\. We find that similar findings to ones forGPT\-4\.1hold for other models\. The most notable cases are Gemini 2\.5 \([Figure˜4\(a\)](https://arxiv.org/html/2606.13254#A2.F4.sf1)\), where persona prompting exhibits the strongest effect and LLaMA 3\.1 \([Figure˜4\(c\)](https://arxiv.org/html/2606.13254#A2.F4.sf3)\), where persona prompting introduces little novel content compared to sampling with higher temperature\. Overall, we believe \(and the results support\) the fact that properly utilizing personas in generation is an emergent capability in LLMs, as evidenced by semantic similarity of novel content being much higher for Gemini and GPT, the two largest models we studied\.

![Refer to caption](https://arxiv.org/html/2606.13254v1/x5.png)\(a\)Gemini 2\.5
![Refer to caption](https://arxiv.org/html/2606.13254v1/x6.png)\(b\)OLMo2 1b
![Refer to caption](https://arxiv.org/html/2606.13254v1/x7.png)\(c\)Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2606.13254v1/x8.png)\(d\)OLMo2 7b

Figure 4:Similarity results across models\.
## Appendix CPre\-training Dataset Analysis\.

We report the full results of reviews and Goodreads overview pages identified in the DCLM component of the OLMo pretraining data mix in[Table˜8](https://arxiv.org/html/2606.13254#A3.T8)\.

Table 8:Estimated review and overview pages per source \(sources with<<10% in both categories omitted\)\.
## Appendix DBook Corpus Details

Throughout our experiments, we used a set of 100 books, 20 representative books chosen from 5 categories\. In[Tables˜9](https://arxiv.org/html/2606.13254#A4.T9)and[10](https://arxiv.org/html/2606.13254#A4.T10)we enumerate the books, their metadata, and the criterion we used to select them for the core book set\.

Table 9:Metadata of books used in experiments \(1–50\), adapted from the Goodreads dataset\(Wan and McAuley,[2018](https://arxiv.org/html/2606.13254#bib.bib4)\)Table 10:Metadata of books used in experiments \(51–100\), adapted from the Goodreads dataset\(Wan and McAuley,[2018](https://arxiv.org/html/2606.13254#bib.bib4)\)