Do Large Language Models Always Tell The Same Stories?

# Do Large Language Models Always Tell The Same Stories?
Source: [https://arxiv.org/html/2606.17350](https://arxiv.org/html/2606.17350)
Hans Ole Hatzel Language Technology Group University of Hamburg thennal10@gmail\.com,hans\.ole\.hatzel@uni\-hamburg\.de

###### Abstract

Recent advances in large language models \(LLMs\) have enabled the generation of high\-quality prose, yet the question of whether these models are capable of generating diverse outputs remains contested\. In this work, we investigate the diversity of LLM\-generated stories through the framework ofnarrative similarity\. Using a contrastive framework and a dataset of human\-written stories and prompts fromr/WritingPrompts, we collect narrative similarity judgments across 10 representative LLMs, utilizing both human evaluations and three different automatic annotation methods\. Our findings reveal a consistent trend: LLM\-generated narratives are consistently more similar to each other than human\-written stories are\. We demonstrate that frontier models in particular converge on a “mean” generic narrative that approximates individual human stories but lacks the collective diversity of human authors\. Finally, we show that common mitigation strategies, including negative prompting and temperature scaling, fail to meaningfully address this homogeneity\.

Do Large Language Models Always Tell The Same Stories?

Thennal D K and Hans Ole HatzelLanguage Technology GroupUniversity of Hamburgthennal10@gmail\.com,hans\.ole\.hatzel@uni\-hamburg\.de

## 1Introduction

With recent advances, large language models \(LLMs\) are being increasingly applied to creative writing tasks, from short stories to poems, with the ability to craft personalized and interactable narratives\(Telekiet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib48); Yuanet al\.,[2022](https://arxiv.org/html/2606.17350#bib.bib56); Tianet al\.,[2024a](https://arxiv.org/html/2606.17350#bib.bib49)\)\. Frontier LLMs are now capable of generating stories and prose that evaluators often find more pleasing compared to human\-written alternatives\(Klingeet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib23); Chakrabarty and Dhillon,[2026](https://arxiv.org/html/2606.17350#bib.bib8)\)\. Despite these capabilities, whether LLM\-generated artifacts are ascreativeordiversein comparison to humans remains an open question, with often contradictory results in the literature\(Gilhooly,[2024](https://arxiv.org/html/2606.17350#bib.bib15); Bellemare\-Pepinet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib3); Luet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib26); Padmakumaret al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib35); Marcoet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib29); Shypulaet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib40)\)\.

![Refer to caption](https://arxiv.org/html/2606.17350v1/x1.png)Figure 1:The distribution of narrative component embedding similarity between pairs of stories, generated by the same model or written by humans, across our dataset\. We note that human\-written stories are consistently less similar to each other than LLM\-generated ones\.We suspect that the breadth of contradictory findings is directly tied to the ambiguity in operationalizing creativity, particularly as it applies to machine generation\. Recent work byLuet al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib28)\)evaluates the methods used in the field and finds that directly judging the creativity of LLM\-generated text is prone to error and inconsistency\. Within this background, our work narrows the focus to the distinct but related question ofnarrative diversity: How similar are LLM\-generated narratives, within and between models as well as in comparison to human\-written ones?

To address this question, we utilize a contrastive annotation framework based on narrative similarity\(Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17)\)\. By asking annotators to determine which of two candidate stories are narratively closer to a reference story, we build a set of paired similarity judgments that allow us to make inferences on the narrative diversity of stories within a given pool\. To build the pools, we collect human\-written prompts and stories fromr/WritingPrompts111[https://www\.reddit\.com/r/WritingPrompts/](https://www.reddit.com/r/WritingPrompts/), and use the prompts to generate corresponding LLM narratives, allowing us to make a direct comparison\.

Through human evaluation and three different automated annotation methods, we conduct a large\-scale exploration of narrative diversity across 10 LLMs, encompassing closed\-source frontier models, smaller open\-source models, and a sequence of post\-training checkpoints\. As illustrated in[Figure˜1](https://arxiv.org/html/2606.17350#S1.F1), our findings reveal a consistent and stark difference in the narrative diversity of humans and LLMs\. Across model families, scale, and post\-training, LLM\-generated narratives are overwhelmingly judged to be more similar to other LLM\-generated narratives, with particularly high rates of similarity within stories generated by the same model\. Closed\-source LLMs are able to emulate human\-written narratives but lack the diversity between them, while smaller LLMs generate narratives that are notably distinct from humans yet still homogeneous\. Finally, we test common mitigation techniques such as negative prompting and temperature scaling, finding that they do little to improve diversity\.222Our code and data are available at:[https://github\.com/thennal10/narrative\-similarity](https://github.com/thennal10/narrative-similarity)\.

## 2Related Work

As generative language models explode in popularity and usage, studies have tried to quantify their creativity in various domains, from problem solving to creative drawing\(Nathet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib33); Tianet al\.,[2024b](https://arxiv.org/html/2606.17350#bib.bib50); Caoet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib6)\)\. We broadly categorize approaches into two groups: rubric\-based evaluation, and automated lexical metrics\.

Several works adapt well\-known psychometric tests and human\-evaluated rubrics for creativity, such as the Alternative Uses Task\(Gilhooly,[2024](https://arxiv.org/html/2606.17350#bib.bib15)\), Torrance Test of Creative Thinking\(Chakrabartyet al\.,[2024](https://arxiv.org/html/2606.17350#bib.bib7)\), and the Divergent Associations Test\(Bellemare\-Pepinet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib3)\)\. Some find that LLMs outperform the average human baselines in these psychometric measures\(Gilhooly,[2024](https://arxiv.org/html/2606.17350#bib.bib15); Bellemare\-Pepinet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib3)\), while others note that they are far from the creativity expressed in expert human writing\(Chakrabartyet al\.,[2024](https://arxiv.org/html/2606.17350#bib.bib7); Gómez\-Rodríguez and Williams,[2023](https://arxiv.org/html/2606.17350#bib.bib16)\)\. While specific psychometric tests have been shown reliable for humans, whether they are equally applicable for LLMs remains contested\(Nakajimaet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib32)\)\. To scale these evaluations, some recent work heavily relies on LLM\-as\-a\-Judge setups\(Paech,[2023](https://arxiv.org/html/2606.17350#bib.bib36); Marcoet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib29)\), but given the subjective nature and difficulty of the task, these approaches are prone to inconsistency and bias\(Luet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib28)\)\.

Other works opt for lexical metrics to quantify creativity\.Luet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib26)\)utilizes a metric derived from n\-gram novelty, finding that professional authors score66\.2%66\.2\\%higher than LLMs, while alignment reduces the score considerably\. However,Padmakumaret al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib35)\)utilizes a similar metric based on unseen and task\-specific n\-grams, finding that post\-training methods actually improve novelty, beyond the average human baseline\.Shypulaet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib40)\)also utilizes a framework measuring a derived semantic diversity, and comes to a similar conclusion regarding the positive effects of post\-training and scale\. In general,Saakyanet al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib39)\)andLuet al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib28)\)caution against n\-gram novelty as a proxy for creativity, finding that it does not match expert evaluations\.

Xuet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib54)\)andTianet al\.\([2024a](https://arxiv.org/html/2606.17350#bib.bib49)\)come closest to directly tackling our question of narrative diversity\.Tianet al\.\([2024a](https://arxiv.org/html/2606.17350#bib.bib49)\)compares LLM\-generated plot summaries with Wikipedia film summaries, and categorizes them into 7 story arcs, finding that LLM\-generated plots overrepresent certain arcs\.Xuet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib54)\)utilize stories fromr/WritingPromptsand Wikipedia plot summaries to define aSui Generisscore that quantifies the repetition in plot elements, finding that LLMs may “echo” particular plot elements\. However, both approaches consider narrative within a narrow scope;Tianet al\.\([2024a](https://arxiv.org/html/2606.17350#bib.bib49)\)only considers a fixed categorization of one aspect of the narrative, andXuet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib54)\)focuses on plot element repetition within 30\-word segments in alternate continuations of stories\. Neither evaluates the narrative as a whole, and both utilize condensed plot summaries rather than the narratives contained within an organic medium\. Our work diverges by directly comparing the overarching narratives that underpin model\-generated stories, and provides broader results on the effects of post\-training and mitigation techniques\.

## 3Methodology

We pose the following central research question: How similar are LLM\-generated narratives, within and across models as well as in comparison to human\-written ones? In order to tackle this directly, we borrow the contrastive format used in a recent shared task onnarrative similarity\(Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17)\)\. Given a pool of stories from a set of generators, we take a triplet: a reference story, and candidate stories A and B\. We format this triplet as an annotation task, asking the annotator to choose whether the narrative of story A or story B is closer to the reference\.

We use the termgeneratorto indicate the entity that created a given story; Humans or the specific LLM\.

We focus on short narratives written with an explicit writing prompt, a choice that aligns our approach with prior work and makes large scale annotations feasible\. The inclusion of a writing prompt provides a starting point for the LLM generation without explicitly dictating the narrative structure, and closer reflects the practical usage of LLMs in story writing\(Yuanet al\.,[2022](https://arxiv.org/html/2606.17350#bib.bib56)\)\.

Our setup provides several distinct advantages: a contrastive format is easier for annotators to judge, and less bound by the specific guidelines for judgment\(Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17)\)\. It is less subjective than prior attempts asking annotators to score creativity based on a rubric, while still measuring the narrative diversity of a generator via narrative similarity judgements\. Finally, restricting ourselves to stories written with a specific writing prompt reduces the difficulty of the judging task further, and provides a fine\-grained comparison\.

### 3\.1Visualization

Our setup measures narrative diversity in a contrastive framework: similarity judgments directly depend on the pool of generators and there is conceptual difference between comparisons where a generator is the reference and where it is a candidate\. Any inference thus requires a broader overview of the data that we could not encompass with a single per\-generator metric\. We instead turn to visualizations, specifically in the form of the heatmap\.

In the heatmap, each rowiicorresponds to the reference generator, and each columnjjcorresponds to the selected candidate generator \(i\.e\. the generator whose story is judged to be more similar\)\. Each elementMi,jM\_\{i,j\}represents the normalized selection rate of generatorjj\. Specifically, given a triplet with the reference story generated byiiand one of the two candidate stories generated byjj,Mi,jM\_\{i,j\}is the proportion of such triplets where the story generated byjjwas found to be more similar to the reference\. In effect, large values in the matrix indicate that the corresponding generators’ stories are more often narratively similar\.

## 4General Setup

In this section, we provide the general outline of our setup, including the models, triplet generation strategy, and dataset we use\.

### 4\.1Dataset

In order to provide a direct comparison between human and LLM\-generated narratives, we require a dataset of story prompts and corresponding stories written by humans\. Following related work, we source stories from Reddit’sr/WritingPromptssubreddit, a forum dedicated to amateur story writing\. Users post writing prompts and other users comment with stories based on the prompt, providing a direct analogy to prompting an LLM\. Users can interact via upvotes and downvotes resulting in a score that indicates a given story’s popularity\.

To quantify the similarity between human\-written stories, we require some triplets where the reference and one of the two candidate stories were written by humans, and so we need two human stories per prompt\. To ensure baseline quality and prevent length bias, we filter for stories between 200 and 300 words with at least 100 upvotes\. To mitigate exposure bias, where authors might alter their submission after reading earlier ones, we exclusively select story pairs posted within 15 minutes of each other\. Following manual cleanup of paratext and duplicates, our dataset comprises 44 prompts, each associated with two pairs of human stories\. This provides a diverse set of story prompts that enables a large set of combinatorial cross\-generator comparisons\.

### 4\.2Models

To evaluate the effects of model architecture, scale, and post\-training, we select 10 total models from three different categories:

- •Closed\-source models:A selection of state\-of\-the\-art models to evaluate frontier LLMs, with Claude Opus 4\.6\(Anthropic,[2026](https://arxiv.org/html/2606.17350#bib.bib2)\), Gemini 3\.1 Pro\(Google DeepMind,[2026](https://arxiv.org/html/2606.17350#bib.bib11)\), GPT\-5\.2\(Singhet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib41)\), and Kimi K2\.5\(Teamet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib46)\)\.
- •Open\-source models:Open\-source models in the∼\\sim30 billion parameter range, with Gemma 3 27B\(Teamet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib45)\), OLMo 3\.1 32B Instruct\(Olmoet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib34)\), and Qwen3 30B\(Yanget al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib55)\)\.
- •OLMo models:The OLMo 3 7B Instruct checkpoints\(Olmoet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib34)\)at various stages of post\-training, specifically after supervised fine\-tuning model \(SFT\), direct preference optimization \(DPO\), and reinforcement learning \(RL\)\. We omit the base model as it did not generate coherent stories and consistently failed to stay within the word limit in early experiments\.

A list of the specific model identifiers are provided in[Appendix˜A](https://arxiv.org/html/2606.17350#A1)\. For each writing prompt in our dataset, we prompt each model twice, creating two generations for each prompt\-model combination\.

### 4\.3Triplet Generation

Within a given pool of stories, we generate triplets such that all stories are used as the reference exactly once, and all combinations of generators are included in the candidates for each reference\. This scheme generates 2112 triplets with 4 generators, and 4400 triplets with 5 generators, and 53,240 triplets with 11 generators\. For details, see[Appendix˜C](https://arxiv.org/html/2606.17350#A3)\.

## 5Annotation

To evaluate narrative similarity across our generated stories, we rely on annotated triplets\. We first conduct human evaluation on a subset of the data, providing preliminary insights\. Subsequently, we explore and validate three automated annotation methods against this baseline, allowing us to scale our evaluation\. We will release all our annotations as well as the code and model checkpoints for our automated annotation methods\.

### 5\.1Human Annotations

Table 1:Inter\-annotator agreement \(Agr\.\) for each triplet category\. The total number of triplets \(N\), and the preferred candidate \(Pref\. \) as well as the percentage of annotations it was preferred in is also provided, except for categories where candidates cannot be distinguished with respect to the reference\.To establish a ground\-truth baseline, we first annotate triplets via human evaluation on 100 sampled triplets\. In addition to humans, we pick two models from each of the model categories described in §[4\.2](https://arxiv.org/html/2606.17350#S4.SS2)in order to include all types of LLMs used in our study while keeping the total generator pool reasonably small\. Specifically, we opt for Opus\-4\.6, Kimi\-K2\.5, Gemma\-3, Qwen3, OLMo\-SFT, and OLMo\-DPO, for a total of 6 models\. In order to ensure sufficient comparisons with human\-written stories, we opt for a stratified sampling procedure that oversamples them\. Annotations were provided by annotators in India and Nigeria, with two independent annotations per triplet\. We utilize the expert guidelines fromHatzelet al\.\([2025](https://arxiv.org/html/2606.17350#bib.bib58)\)with minimal modification\. Annotators were also asked to report their confidence in their selection on a scale of 1 to 5\. Additional details are provided in[Appendix˜D](https://arxiv.org/html/2606.17350#A4)\.

As there are not enough annotations for a full heatmap, we abstract specific generators within the triplets and aggregate them into triplet categories of the form\(X, Y, Z\), whereX,Y, andZare generators\. We useHto represent human generated stories andMkfor the k\-th unique generator model in that triplet\. We do not distinguish by the ordering of the candidates, e\.g\.\(M1, H, M2\)=\(M1, M2, H\)\. For details on this notation scheme, see[Section˜D\.3](https://arxiv.org/html/2606.17350#A4.SS3)\.

Table[1](https://arxiv.org/html/2606.17350#S5.T1)reports the inter\-rater agreement between the two annotator cohorts, as well as the preferred candidate in each category\. Overall agreement is high at86%86\\%, and we note that agreement remains high for categories with repeated generators or a human\-generated candidate\. Conversely, the agreement is particularly low when the candidate stories are generated by two different models, distinct from the generator, i\.e\.\(H, M1, M2\)and\(M1, M2, M3\)\. This indicates that these categories are more ambiguous and subjective, and annotation is not as reliable\. We also note that this also aligns with our hypothesis: LLMs generate similar stories, and thus are harder to distinguish between as candidates to a reference that is completely distinct \(human\) or equally similar \(another LLM\)\.

The preferred candidate for each category further complements our hypothesis\. When evaluated against model\-generated reference, a model candidate is preferred over a human candidate91\.1%91\.1\\%of the time if the models differ, and100%100\\%of the time if the model is the same\. Even with a human reference and a human candidate, a model candidate is preferred72\.7%72\.7\\%of the time, indicating that human stories are less similar to each other than it is to a model story\.

Regardless, with only 100 triplets, broader inferences and comparisons between specific generators are difficult to make, and thus we turn to automated annotation\.

### 5\.2Automated Annotation

Due to the number of annotations required for a comprehensive overview, we require automated annotation\. Inspired by the approaches outlined inHatzelet al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib17)\), we investigate three distinct methods:

- •LLM\-as\-a\-Judge: Utilizing GPT\-5 to directly annotate triplets based on a given categorization of narrative similarity\(Paech,[2023](https://arxiv.org/html/2606.17350#bib.bib36); Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17)\)\. We also prompt for a confidence score in the judgment on a scale of 1 to 5, as with human annotation\.
- •Narrative Component Embedding: Decomposing stories into narrative components \(abstract themes, actions, and outcomes\) via GPT\-5, embedding them using Gemini Embedding, and using the cosine similarity to annotate triplets\(Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17); Leeet al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib24)\)\.
- •Preference Model: A lightweight preference model based on the Bradley\-Terry formulation\(Bradley and Terry,[1952](https://arxiv.org/html/2606.17350#bib.bib5)\)trained on synthetically generated annotations, with Qwen3 1\.7B as the base model\(Yanget al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib55)\)\.

The specific prompts used and other additional details for each setup are given in[Appendix˜E](https://arxiv.org/html/2606.17350#A5)\.

Table 2:Average agreement with human annotators of the LLM\-as\-a\-Judge \(LLM\), narrative component embedding \(Emb\.\), and preference model \(PM\) annotation methods across the different triplet categories\.![Refer to caption](https://arxiv.org/html/2606.17350v1/x2.png)Figure 2:Similarity heatmap showcasing the normalized selection rate for all models annotated via narrative component embedding\. The rows indicate the reference and the columns indicate the selected candidate generator, as detailed in[Section˜3\.1](https://arxiv.org/html/2606.17350#S3.SS1)\.![Refer to caption](https://arxiv.org/html/2606.17350v1/x3.png)Figure 3:Similarity heatmap of triplets annotated by the LLM judge and with a confidence filter of 5 within the closed\-source, open\-source, and OLMo generator pools\.We evaluate the agreement of all three automated annotation methods against the human annotations, with the results compiled in[Table˜2](https://arxiv.org/html/2606.17350#S5.T2)\. LLM\-as\-a\-Judge and narrative component embedding perform equally well, with the latter performing slightly better overall at82%82\\%agreement\. The preference model performs notably worse at77%77\\%agreement\. We note that the two categories in which judges performed the worst are the same categories where inter\-annotator agreement is notably lower\. The preference model’s lower overall agreement is also largely a result of poorer performance in these categories\.

Furthermore, we find that the LLM judge’s self\-reported confidence serves as a reliable indicator of accuracy\. Overall agreement reaches92\.8%92\.8\\%when the LLM reports maximum confidence \(a score of 5\), suggesting a confidence cutoff can effectively filter reliable annotations \(a full heatmap and confidence breakdown is provided in[Section˜D\.4](https://arxiv.org/html/2606.17350#A4.SS4)\)\.

Given that all three methods largely performed well, we opt for each in different contexts: for large\-scale comparisons with the entire model pool, we use the narrative component embedding method, for a more detailed comparison between specific generators, we use the LLM\-as\-a\-Judge method, and for additional mitigation experiments where efficient repeated annotations are required, we use the preference model\.

## 6Narrative Similarity

Using all three automated annotation methods, we conduct a large\-scale evaluation of narrative similarity across model pools\.

### 6\.1All Models

First, we use narrative component embedding to annotate triplets generated for all 11 generators \(10 models and humans\)\. The resulting heatmap is provided in[Figure˜2](https://arxiv.org/html/2606.17350#S5.F2)\.

We note a universal trend: LLM\-generated stories were judged to be substantially more similar to other LLM\-generated stories, with only a small fraction \(between9\.2%9\.2\\%and16\.8%16\.8\\%\) of LLM\-generated stories judged to be more similar to a human\-written one\. Additionally, self\-similarity \(the diagonal elements\) is high across all models, and in particular the open and closed source models: almost all in the80%80\\%to90%90\\%range with the exception of Kimi\-K2\.5 at73\.9%73\.9\\%\. In contrast, human self\-similarity largely remains low at30\.8%30\.8\\%\.

Key TakeawayLLM\-generated stories are narratively more similar to each other than human\-written stories are\.

The OLMo 7B models follow the same general trends, but diverge in notable ways\. In particular, self\-similarity is not substantially higher than inter\-model similarity, while similarity with other models is noticeably low: a non\-OLMo 7B model\-generated story is judged to be more similar to an OLMo 7B model\-generated one26\.6%26\.6\\%to47\.4%47\.4\\%of the time\. Given that the OLMo 7B models are all checkpoints from the same base model, this is not a surprising result\. Nevertheless, we note that out of all models, the OLMo SFT model has the lowest self\-similarity at62\.6%62\.6\\%\.

### 6\.2Within Model Categories

We take a more fine\-grained look at each model category using the LLM judge\. The closed source, open source, and OLMo models are considered separately in their own generator pools\. Within each pool, we select and annotate triplets, filtering out all annotated triplets with a confidence score less than 5, as lower\-confidence annotations are significantly less reliable\. The resulting heatmaps are provided in[Figure˜3](https://arxiv.org/html/2606.17350#S5.F3)\.

As before in[Section˜6\.1](https://arxiv.org/html/2606.17350#S6.SS1), we observe that LLM\-generated stories are judged to be substantially more similar to other LLM\-generated stories than to human\-generated ones, but with starker difference: across all three pools, model\-human similarity is between0\.0%0\.0\\%and3\.9%3\.9\\%\. Human self\-similarity is also extremely low in the closed source pool at6\.8%6\.8\\%, but increases to21\.6%21\.6\\%in the open source pool, and to50\.0%50\.0\\%in the OLMo pool\.

We may interpret this as a trend towards converging on a “mean” narrative with increasing scale and post\-training\. Large closed\-source models generate stories which approximate human\-generated narratives better but lack the diversity found between different human narratives\. Therefore, a human\-generated reference is likelier to be similar to a model\-generated story than they are to another human\-generated story\. In contrast, the OLMo series generates narratives which are distinctly different from human\-generated narratives, and so a human\-generated reference is equally dissimilar to both model and human\-generated candidates\. We again note that the OLMo SFT model has the lowest self\-similarity among all models at66\.7%66\.7\\%, while the DPO and RL models have both high self\-similarity and inter\-modal similarity, forming a distinct visual block\.

Key TakeawayWith increasing scale and post\-training, LLMs converge on generic narratives, closer to individual human narratives while lacking the diversity found across them\.

We note that similar results were observed without the confidence filter as well, as shown in[Section˜F\.1](https://arxiv.org/html/2606.17350#A6.SS1)\. We also annotate the same model pools with the preference model in[Section˜F\.2](https://arxiv.org/html/2606.17350#A6.SS2), showing that there is little difference between the results of the two annotation methods\.

![Refer to caption](https://arxiv.org/html/2606.17350v1/x4.png)Figure 4:Similarity heatmap of triplets annotated by the preference model within the closed\-source, open\-source, and OLMo generator pools, and with stories generated sequentially \(each model is provided all prior generated stories\)\.![Refer to caption](https://arxiv.org/html/2606.17350v1/x5.png)Figure 5:Similarity heatmap of triplets annotated by the preference model, within the OLMo pool under a selected range of temperatures\.

## 7Mitigation Strategies

Several mitigation strategies exist in the literature to increase diversity, including prompting and decoding methods\(Zhanget al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib57); Tianet al\.,[2024a](https://arxiv.org/html/2606.17350#bib.bib49); Parket al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib37)\)\. We test whether these have any effect on narrative diversity using the preference model\.

### 7\.1Sequential Negative Prompting

A common and simple mitigation strategy is to generate stories sequentially, providing the previous story as context and prompting the model to generate a different story\(Zhanget al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib57)\)\. We test a more rigorous version of this strategy for all three pools: instead of only providing the previous stories as context for that specific model, we generate stories within the pool in a round\-robin fashion, appendingallprior model\-generated stories \(including other models within the pool\) and explicitly prompting the model to generate a story that is as different as possible\. We annotate the resulting triplets with the preference model, and the resulting heatmap is provided in[Figure˜4](https://arxiv.org/html/2606.17350#S6.F4)\. We note that while self\-similarity is reduced, it is still largely higher than inter\-model similarity, and LLM\-generated stories remain substantially more similar to each other than to human\-generated stories, with little improvement in overall diversity\. We also note that GPT\-5\.2\-generated stories are particularly likely to be judged more similar regardless of the reference, indicating that this strategy’s effectiveness is also dependent on the specific model used\.

Key TakeawayEven with explicit negative prompting against prior generated stories, LLM\-generated narratives remain homogeneous\.

### 7\.2Sampling Parameters

We test the effects of two common sampling parameters that are utilized to make model outputs more diverse: temperature and top\-P\(Holtzmanet al\.,[2020](https://arxiv.org/html/2606.17350#bib.bib18); Ackleyet al\.,[1985](https://arxiv.org/html/2606.17350#bib.bib1)\)\. Using the preference model and OLMo pool, we generate stories under all combinations of temperatureT∈\{0\.2,0\.4,0\.6,0\.8,1\.0\}T\\in\\\{0\.2,0\.4,0\.6,0\.8,1\.0\\\}and top\-PP∈\{0\.85,0\.9,0\.95,1\.0\}P\\in\\\{0\.85,0\.9,0\.95,1\.0\\\}\. Surprisingly, we find that top\-P has no consistent impact\. For brevity, we showcase the heatmaps of three selected temperature values \(0\.2,0\.6,1\.00\.2,0\.6,1\.0\) all with top\-P of0\.90\.9in[Figure˜5](https://arxiv.org/html/2606.17350#S6.F5)\. The full set of heatmaps can be found in[Section˜F\.3](https://arxiv.org/html/2606.17350#A6.SS3)\.

We note that the only model that seems to be appreciably affected by sampling parameters is SFT, with self\-similarity decreasing considerably from99\.299\.2to76\.976\.9as temperature goes from0\.20\.2to1\.01\.0\. However, SFT\-generated stories are still rarely judged to be more similar to a human\-generated story, even with a temperature of1\.01\.0\. These results also support prior work, which have found that post\-training leads to a collapse in the logit probability calibration, making temperature scaling ineffective\(Tanet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib44); Xieet al\.,[2024](https://arxiv.org/html/2606.17350#bib.bib53)\)\.

Key TakeawaySampling parameter scaling can reduce self\-similarity in non\-aligned models but have little effect in preference\-tuned models\.

## 8Conclusion

Our study provides a novel and robust operationalization of creativity through the lens of narrative diversity\. By employing a contrastive similarity framework within the domain of writing prompt stories, we provide a fine\-grained comparison of narratives within and across humans and LLM models\. Utilizing both human and automated annotation methods, our findings reveal that across a wide range of models, LLMs generate narratives that are consistently more similar to each other than human\-written stories\. We also find that post\-training and increasing scale both contribute to this lack of diversity, with frontier LLMs in particular generating generic “mean” narratives that are closer to human\-generated stories but lacking the diversity found between different human\-generated stories\.

We also test common mitigation strategies to improve diversity, and find that none of them show meaningful improvements, pointing to a clear question for future work\. Within this conception of creativity, we conclude that today’s LLMs are not creative storytellers, and this limitation cannot be ignored as they see increasing use in creative applications\.

## Limitations

As we source our writing prompts and human\-written stories from Reddit in the timespan of 2014 to 2024, the training data of the models we evaluate may be contaminated\. However, we posit that contamination is unlikely to be a relevant factor in our findings, as we would expect high similarity between human\-written stories and model\-generated stories if the models had memorized or otherwise utilized elements of the human\-written stories\.

Additionally, when training the preference model, we exclude the specific stories in human\-annotated triplets from the training data\. However, the stories are still generated by the same models that we evaluate with, and with largely the same set of writing prompts\. Thus, the preference model may not be generalizable beyond these writing prompts or model pools\.

Finally, while we test the effects of sampling parameters on diversity, we do not test decoding methods such as avoidance decoding\(Parket al\.,[2025](https://arxiv.org/html/2606.17350#bib.bib37)\), which may have a stronger effect\. Regardless, as we are primarily interested in narrative\-level diversity, we hypothesize that token\-level decoding methods are unlikely to be an effective mitigation strategy\.

## Ethical Considerations

As our work is largely concerned with analysis of existing LLMs, we do not foresee any direct ethical concerns with our study\. The artifacts we release, including the annotations, code, and model checkpoints, are all explicitly designed for narrative similarity evaluation; they do not contain sensitive, toxic, or personally identifiable information, nor are they suited for generating harmful content\. Thus, we do not anticipate any potential for misuse\.

## References

- D\. H\. Ackley, G\. E\. Hinton, and T\. J\. Sejnowski \(1985\)A learning algorithm for boltzmann machines\.Cognitive Science9\(1\),pp\. 147–169\.External Links:ISSN 0364\-0213,[Document](https://dx.doi.org/10.1016/S0364-0213%2885%2980012-4)Cited by:[§7\.2](https://arxiv.org/html/2606.17350#S7.SS2.p1.4)\.
- Anthropic \(2026\)Claude Opus 4\.6 System Card\.External Links:[Link](https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf)Cited by:[1st item](https://arxiv.org/html/2606.17350#S4.I1.i1.p1.1)\.
- A\. Bellemare\-Pepin, F\. Lespinasse, P\. Thölke, Y\. Harel, K\. Mathewson, J\. A\. Olson, Y\. Bengio, and K\. Jerbi \(2026\)Divergent creativity in humans and large language models\.Scientific Reports16\(1\),pp\. 1279\.External Links:ISSN 2045\-2322,[Document](https://dx.doi.org/10.1038/s41598-025-25157-3)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank Analysis of Incomplete Block Designs: I\. The Method of Paired Comparisons\.Biometrika39\(3/4\),pp\. 324–345\.External Links:2334029,ISSN 0006\-3444,[Document](https://dx.doi.org/10.2307/2334029)Cited by:[§E\.3](https://arxiv.org/html/2606.17350#A5.SS3.p1.5),[3rd item](https://arxiv.org/html/2606.17350#S5.I1.i3.p1.1)\.
- Q\. Cao, X\. Wang, Y\. Yuan, Y\. Liu, F\. Luo, and R\. Song \(2026\)Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator\.InThe Fourteenth International Conference on Learning Representations,Rio de Janeiro, Brazil\.External Links:[Link](https://openreview.net/forum?id=WHrmlyQEM7)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p1.1)\.
- T\. Chakrabarty and P\. S\. Dhillon \(2026\)Can Good Writing Be Generative? Expert\-Level AI Writing Emerges through Fine\-Tuning on High Quality Books\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems,CHI ’26,New York, NY, USA\.External Links:ISBN 9798400722783,[Link](https://doi.org/10.1145/3772318.3791276),[Document](https://dx.doi.org/10.1145/3772318.3791276)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1)\.
- T\. Chakrabarty, P\. Laban, D\. Agarwal, S\. Muresan, and C\. Wu \(2024\)Art or Artifice? Large Language Models and the False Promise of Creativity\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,CHI ’24,New York, NY, USA\.External Links:ISBN 9798400703300,[Link](https://doi.org/10.1145/3613904.3642731),[Document](https://dx.doi.org/10.1145/3613904.3642731)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- K\. Gilhooly \(2024\)AI vs humans in the AUT: Simulations to LLMs\.Journal of Creativity34\(1\),pp\. 100071\.External Links:ISSN 2713\-3745,[Document](https://dx.doi.org/10.1016/j.yjoc.2023.100071)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- C\. Gómez\-Rodríguez and P\. Williams \(2023\)A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 14504–14528\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.966/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.966)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- Google DeepMind \(2026\)Gemini 3\.1 Pro Model Card\.External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[1st item](https://arxiv.org/html/2606.17350#S4.I1.i1.p1.1)\.
- H\. O\. Hatzel, E\. Artemova, H\. Stiemer, N\. Fedorova, E\. Gius, and C\. Biemann \(2025\)Narrative Similarity – Annotation Guidelines\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.16537509),[Link](https://doi.org/10.5281/zenodo.16537509)Cited by:[§5\.1](https://arxiv.org/html/2606.17350#S5.SS1.p1.1)\.
- H\. O\. Hatzel, E\. Artemova, H\. P\. Stiemer, E\. Gius, and C\. Biemann \(2026\)SemEval\-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning\.External Links:2604\.21782,[Link](https://arxiv.org/abs/2604.21782)Cited by:[§E\.1](https://arxiv.org/html/2606.17350#A5.SS1.p1.1),[§E\.2](https://arxiv.org/html/2606.17350#A5.SS2.p1.1),[§1](https://arxiv.org/html/2606.17350#S1.p3.1),[§3](https://arxiv.org/html/2606.17350#S3.p1.1),[§3](https://arxiv.org/html/2606.17350#S3.p4.1),[1st item](https://arxiv.org/html/2606.17350#S5.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2606.17350#S5.I1.i2.p1.1),[§5\.2](https://arxiv.org/html/2606.17350#S5.SS2.p1.1)\.
- A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi \(2020\)The Curious Case of Neural Text Degeneration\.InThe Eighth International Conference on Learning Representations,Addis Ababa, Ethiopia\.External Links:[Link](https://openreview.net/forum?id=rygGQyrFvH)Cited by:[§7\.2](https://arxiv.org/html/2606.17350#S7.SS2.p1.4)\.
- International Labour Organization \(2024\)Global Wage Report 2024–25: Is wage inequality decreasing globally?\.International Labour Office,Geneva\.External Links:ISBN 9789220409770,[Link](https://www.ilo.org/publications/flagship-reports/global-wage-report-2024-25-wage-inequality-decreasing-globally)Cited by:[§D\.2](https://arxiv.org/html/2606.17350#A4.SS2.p3.1)\.
- X\. Klinge, S\. Ortlieb, and A\. Koller \(2026\)LLMs generate kitsch\.arXiv preprint arXiv:2604\.25929\.External Links:2604\.25929Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1)\.
- J\. Lee, F\. Chen, S\. Dua, D\. Cer, M\. Shanbhogue, I\. Naim, G\. H\. Ábrego, Z\. Li, K\. Chen, H\. S\. Vera, X\. Ren, S\. Zhang, D\. Salz, M\. Boratko, J\. Han, B\. Chen, S\. Huang, V\. Rao, P\. Suganthan, F\. Han, A\. Doumanoglou, N\. Gupta, F\. Moiseev, C\. Yip, A\. Jain, S\. Baumgartner, S\. Shahi, F\. P\. Gomez, S\. Mariserla, M\. Choi, P\. Shah, S\. Goenka, K\. Chen, Y\. Xia, K\. Chen, S\. M\. K\. Duddu, Y\. Chen, T\. Walker, W\. Zhou, R\. Ghiya, Z\. Gleicher, K\. Gill, Z\. Dong, M\. Seyedhosseini, Y\. Sung, R\. Hoffmann, and T\. Duerig \(2025\)Gemini Embedding: Generalizable Embeddings from Gemini\.arXiv\.External Links:2503\.07891,[Document](https://dx.doi.org/10.48550/arXiv.2503.07891)Cited by:[2nd item](https://arxiv.org/html/2606.17350#S5.I1.i2.p1.1)\.
- L\. Lu, M\. Liu, P\. C\. Lu, Y\. Tian, S\. Sun, and N\. Peng \(2026\)Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 6329–6352\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.297),ISBN 979\-8\-89176\-380\-7Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p2.1),[§2](https://arxiv.org/html/2606.17350#S2.p2.1),[§2](https://arxiv.org/html/2606.17350#S2.p3.1)\.
- X\. Lu, M\. Sclar, S\. Hallinan, N\. Mireshghallah, J\. Liu, S\. Han, A\. Ettinger, L\. Jiang, K\. Chandu, N\. Dziri, and Y\. Choi \(2025\)AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text\.InThe Thirteenth International Conference on Learning Representations,Singapore\.External Links:[Link](https://openreview.net/forum?id=ilOEOIqolQ)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p3.1)\.
- G\. Marco, L\. Rello, and J\. Gonzalo \(2025\)Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 6552–6570\.External Links:[Link](https://aclanthology.org/2025.coling-main.437/)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- K\. Nakajima, J\. Zuiderveld, and S\. Pezzelle \(2026\)Beyond Divergent Creativity: A Human\-Based Evaluation of Creativity in Large Language Models\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 2639–2660\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.138/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.138),ISBN 979\-8\-89176\-386\-9Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p2.1)\.
- S\. S\. Nath, G\. del Cuvillo y Schršder, and C\. Stevenson \(2025\)Pencils to Pixels: A Systematic Study of Creative Drawings across Children, Adults and AI\.Proceedings of the Annual Meeting of the Cognitive Science Society47\(0\)\.External Links:[Link](https://escholarship.org/uc/item/1rq6h8xg)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p1.1)\.
- T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2026\)Olmo 3\.arXiv\.External Links:2512\.13961,[Document](https://dx.doi.org/10.48550/arXiv.2512.13961)Cited by:[2nd item](https://arxiv.org/html/2606.17350#S4.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2606.17350#S4.I1.i3.p1.1)\.
- V\. Padmakumar, C\. Yueh\-Han, J\. Pan, V\. Chen, and H\. He \(2026\)Measuring LLM Novelty As The Frontier Of Original And High\-Quality Output\.InThe Fourteenth International Conference on Learning Representations,Rio de Janeiro, Brazil\.External Links:[Link](https://openreview.net/forum?id=i7QNKZioN6)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p3.1)\.
- S\. J\. Paech \(2023\)EQ\-Bench: An Emotional Intelligence Benchmark for Large Language Models\.arXiv\.External Links:2312\.06281,[Document](https://dx.doi.org/10.48550/arXiv.2312.06281)Cited by:[§E\.1](https://arxiv.org/html/2606.17350#A5.SS1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p2.1),[1st item](https://arxiv.org/html/2606.17350#S5.I1.i1.p1.1)\.
- K\. Park, N\. Yang, and K\. Jung \(2025\)Avoidance Decoding for Diverse Multi\-Branch Story Generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 7489–7505\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.381/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.381),ISBN 979\-8\-89176\-332\-6Cited by:[§7](https://arxiv.org/html/2606.17350#S7.p1.1),[Limitations](https://arxiv.org/html/2606.17350#Sx1.p3.1)\.
- A\. Saakyan, N\. Kim, S\. Muresan, and T\. Chakrabarty \(2026\)Death of the Novel\(ty\): Beyond n\-Gram Novelty as a Metric for Textual Creativity\.InThe Fourteenth International Conference on Learning Representations,Rio de Janeiro, Brazil\.External Links:[Link](https://openreview.net/forum?id=z2idLjqzBe)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p3.1)\.
- A\. Shypula, S\. Li, B\. Zhang, V\. Padmakumar, K\. Yin, and O\. Bastani \(2025\)Evaluating the diversity and quality of LLM generated content\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=O7bF6nlSOD)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p3.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. J\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. J\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. J\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. d\. A\. B\. Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. J\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Q\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. Wang \(2026\)OpenAI GPT\-5 System Card\.arXiv\.External Links:2601\.03267,[Document](https://dx.doi.org/10.48550/arXiv.2601.03267)Cited by:[1st item](https://arxiv.org/html/2606.17350#S4.I1.i1.p1.1)\.
- W\. Tan, F\. Parascandolo, E\. Sangineto, J\. Ju, Z\. Luo, Q\. Cao, R\. Cucchiara, R\. Song, and J\. Luan \(2026\)Restoring Exploration after Post\-Training: Latent Exploration Decoding for Large Reasoning Models\.arXiv\.External Links:2602\.01698,[Document](https://dx.doi.org/10.48550/arXiv.2602.01698)Cited by:[§7\.2](https://arxiv.org/html/2606.17350#S7.SS2.p2.5)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. J\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 Technical Report\.arXiv\.External Links:2503\.19786,[Document](https://dx.doi.org/10.48550/arXiv.2503.19786)Cited by:[2nd item](https://arxiv.org/html/2606.17350#S4.I1.i2.p1.1)\.
- K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. H\. Cai, Y\. Cao, Y\. Charles, H\. S\. Che, C\. Chen, G\. Chen, H\. Chen, J\. Chen, J\. Chen, J\. Chen, J\. Chen, K\. Chen, L\. Chen, R\. Chen, X\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, D\. Cheng, M\. Chu, J\. Cui, J\. Deng, M\. Diao, H\. Ding, M\. Dong, M\. Dong, Y\. Dong, Y\. Dong, A\. Du, C\. Du, D\. Du, L\. Du, Y\. Du, Y\. Fan, S\. Fang, Q\. Feng, Y\. Feng, G\. Fu, K\. Fu, H\. Gao, T\. Gao, Y\. Ge, S\. Geng, C\. Gong, X\. Gong, Z\. Gongque, Q\. Gu, X\. Gu, Y\. Gu, L\. Guan, Y\. Guo, X\. Hao, W\. He, W\. He, Y\. He, C\. Hong, H\. Hu, J\. Hu, Y\. Hu, Z\. Hu, K\. Huang, R\. Huang, W\. Huang, Z\. Huang, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Jing, G\. Lai, A\. Li, C\. Li, C\. Li, F\. Li, G\. Li, G\. Li, H\. Li, H\. Li, J\. Li, J\. Li, J\. Li, L\. Li, M\. Li, W\. Li, W\. Li, X\. Li, X\. Li, Y\. Li, Y\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, W\. Liao, J\. Lin, X\. Lin, Z\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, L\. Liu, S\. Liu, S\. Liu, S\. Liu, T\. Liu, T\. Liu, W\. Liu, X\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, Z\. Liu, E\. Lu, H\. Lu, Z\. Lu, J\. Luo, T\. Luo, Y\. Luo, L\. Ma, Y\. Ma, S\. Mao, Y\. Mei, X\. Men, F\. Meng, Z\. Meng, Y\. Miao, M\. Ni, K\. Ouyang, S\. Pan, B\. Pang, Y\. Qian, R\. Qin, Z\. Qin, J\. Qiu, B\. Qu, Z\. Shang, Y\. Shao, T\. Shen, Z\. Shen, J\. Shi, L\. Shi, S\. Shi, F\. Song, P\. Song, T\. Song, X\. Song, H\. Su, J\. Su, Z\. Su, L\. Sui, J\. Sun, J\. Sun, T\. Sun, F\. Sung, Y\. Tai, C\. Tang, H\. Tang, X\. Tang, Z\. Tang, J\. Tao, S\. Teng, C\. Tian, P\. Tian, A\. Wang, B\. Wang, C\. Wang, C\. Wang, C\. Wang, D\. Wang, D\. Wang, D\. Wang, F\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, J\. Wang, J\. Wang, J\. Wang, K\. Wang, L\. Wang, Q\. Wang, S\. Wang, S\. Wang, S\. Wang, W\. Wang, X\. Wang, X\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, C\. Wei, M\. Wei, C\. Wen, Z\. Wen, C\. Wu, H\. Wu, J\. Wu, R\. Wu, W\. Wu, Y\. Wu, Y\. Wu, Y\. Wu, Z\. Wu, C\. Xiao, J\. Xie, X\. Xie, Y\. Xie, Y\. Xin, B\. Xing, B\. Xu, J\. Xu, J\. Xu, J\. Xu, L\. H\. Xu, L\. Xu, S\. Xu, W\. Xu, X\. Xu, X\. Xu, Y\. Xu, Y\. Xu, Y\. Xu, Z\. Xu, Z\. Xu, J\. Yan, Y\. Yan, G\. Yang, H\. Yang, J\. Yang, K\. Yang, N\. Yang, R\. Yang, X\. Yang, X\. Yang, Y\. Yang, Y\. Yang, Y\. Yang, Z\. Yang, Z\. Yang, Z\. Yang, H\. Yao, D\. Ye, W\. Ye, Z\. Ye, B\. Yin, C\. Yu, L\. Yu, T\. Yu, T\. Yu, E\. Yuan, M\. Yuan, X\. Yuan, Y\. Yue, W\. Zeng, D\. Zha, H\. Zhan, D\. Zhang, H\. Zhang, J\. Zhang, P\. Zhang, Q\. Zhang, R\. Zhang, X\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Z\. Zhang, C\. Zhao, F\. Zhao, J\. Zhao, S\. Zhao, X\. Zhao, Y\. Zhao, Z\. Zhao, H\. Zheng, R\. Zheng, S\. Zheng, T\. Zheng, J\. Zhong, L\. Zhong, W\. Zhong, M\. Zhou, R\. Zhou, X\. Zhou, Z\. Zhou, J\. Zhu, L\. Zhu, X\. Zhu, Y\. Zhu, Z\. Zhu, J\. Zhuang, W\. Zhuang, Y\. Zou, and X\. Zu \(2026\)Kimi K2\.5: Visual Agentic Intelligence\.arXiv\.External Links:2602\.02276,[Document](https://dx.doi.org/10.48550/arXiv.2602.02276)Cited by:[1st item](https://arxiv.org/html/2606.17350#S4.I1.i1.p1.1)\.
- M\. Teleki, V\. Bengali, X\. Dong, S\. T\. Janjur, H\. Liu, T\. Liu, C\. Wang, T\. Liu, Y\. Zhang, F\. Shipman, and J\. Caverlee \(2025\)A Survey on LLMs for Story Generation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 13954–13966\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.750),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1)\.
- Y\. Tian, T\. Huang, M\. Liu, D\. Jiang, A\. Spangher, M\. Chen, J\. May, and N\. Peng \(2024a\)Are Large Language Models Capable of Generating Human\-Level Narratives?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17659–17681\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.978)Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§2](https://arxiv.org/html/2606.17350#S2.p4.1),[§7](https://arxiv.org/html/2606.17350#S7.p1.1)\.
- Y\. Tian, A\. Ravichander, L\. Qin, R\. Le Bras, R\. Marjieh, N\. Peng, Y\. Choi, T\. Griffiths, and F\. Brahman \(2024b\)MacGyver: Are Large Language Models Creative Problem Solvers?\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 5303–5324\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.297)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p1.1)\.
- J\. Xie, A\. S\. Chen, Y\. Lee, E\. Mitchell, and C\. Finn \(2024\)Calibrating Language Models with Adaptive Temperature Scaling\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 18128–18138\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1007)Cited by:[§7\.2](https://arxiv.org/html/2606.17350#S7.SS2.p2.5)\.
- W\. Xu, N\. Jojic, S\. Rao, C\. Brockett, and B\. Dolan \(2025\)Echoes in AI: Quantifying lack of plot diversity in LLM outputs\.Proceedings of the National Academy of Sciences122\(35\),pp\. e2504966122\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2504966122)Cited by:[§2](https://arxiv.org/html/2606.17350#S2.p4.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 Technical Report\.arXiv\.External Links:2505\.09388,[Document](https://dx.doi.org/10.48550/arXiv.2505.09388)Cited by:[2nd item](https://arxiv.org/html/2606.17350#S4.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2606.17350#S5.I1.i3.p1.1)\.
- A\. Yuan, A\. Coenen, E\. Reif, and D\. Ippolito \(2022\)Wordcraft: Story Writing With Large Language Models\.InProceedings of the 27th International Conference on Intelligent User Interfaces,IUI ’22,New York, NY, USA,pp\. 841–852\.External Links:[Document](https://dx.doi.org/10.1145/3490099.3511105),ISBN 978\-1\-4503\-9144\-3Cited by:[§1](https://arxiv.org/html/2606.17350#S1.p1.1),[§3](https://arxiv.org/html/2606.17350#S3.p3.1)\.
- J\. Zhang, S\. Yu, D\. Chong, A\. Sicilia, M\. R\. Tomz, C\. D\. Manning, and W\. Shi \(2025\)Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity\.arXiv\.External Links:2510\.01171,[Document](https://dx.doi.org/10.48550/arXiv.2510.01171)Cited by:[§7\.1](https://arxiv.org/html/2606.17350#S7.SS1.p1.1),[§7](https://arxiv.org/html/2606.17350#S7.p1.1)\.

## Appendix AModels

We utilize the following list of models throughout our experiments, with the corresponding OpenRouter API endpoint \(for the proprietary models\) or the Hugging Face repository name \(for all other models\) provided:

#### Closed\-Source Pool

- •Opus\-4\.6:anthropic/claude\-opus\-4\.6
- •Gemini\-3\.1:google/gemini\-3\.1\-pro\-preview
- •GPT\-5\.2:openai/gpt\-5\.2
- •Kimi\-K2\.5:moonshotai/kimi\-k2\.5

#### Open\-Source Pool

- •Gemma\-3:google/gemma\-3\-27b\-it
- •OLMo3\.1: allenai/Olmo\-3\.1\-32B\-Instruct
- •OLMo3\.1\-Think:allenai/Olmo\-3\.1\-32B\-Think
- •Qwen3:Qwen/Qwen3\-30B\-A3B
- •Qwen3\-Think:Qwen/Qwen3\-30B\-A3B\-thinking

#### OLMo 7B Series

- •Base:allenai/Olmo\-3\-1025\-7B
- •SFT:allenai/Olmo\-3\-7B\-Instruct\-SFT
- •DPO:allenai/Olmo\-3\-7B\-Instruct\-DPO
- •RL:allenai/Olmo\-3\-7B\-Instruct

We use GPT\-5 \(gpt\-5\) as the judge for all LLM\-as\-a\-Judge evaluations\. Unless specified otherwise, all stories are generated with the default sampling parameters for each model, or with a temperature and top\-P of1\.01\.0when the default is not specified\. All prompts used to generate stories are provided in[Figure˜16](https://arxiv.org/html/2606.17350#A8.F16)\.

## Appendix BLicenses

### B\.1Dataset

The stories and prompts fromr/WritingPromptsare publicly available on Reddit, and the original authors retain the copyright to their respective texts\. In compliance with Reddit’s Terms of Service regarding data redistribution, we do not release the raw text of the stories or prompts, but provide a complete list of post and comment IDs used in our dataset\. All annotations are released under the CC BY\-SA 4\.0 License, and all code and model checkpoints are released under the MIT License\.

### B\.2Models

Our evaluation utilizes a combination of open\-source and proprietary LLMs\. The proprietary models are subject to their respective commercial API Terms of Service\. The open\-source models are subject to the following licenses, which provide permission for research and non\-commercial use:

- •OLMo 3 and 3\.1 Series:Apache 2\.0 License\.
- •Gemma 3:Gemma Terms of Use\.333https://ai\.google\.dev/gemma/terms
- •Qwen3:Apache 2\.0 License\.

## Appendix CTriplet Generation

For a pool of storiesSS, we construct triplets\(sr,sa,sb\)\(s\_\{r\},s\_\{a\},s\_\{b\}\)by iterating through every story in the pool to serve as a reference story,sr∈Ss\_\{r\}\\in S\. For a givensrs\_\{r\}associated with promptpp, we iterate through all combinations of generators\{a,b\}∈\(G2\)\\\{a,b\\\}\\in\{G\\choose 2\}\. As there are two stories for each generator and prompt, we randomly select one foraaandbb, providing ussas\_\{a\}andsbs\_\{b\}\. To ensure the triplet consists of distinct stories, we enforcesa≠srs\_\{a\}\\neq s\_\{r\}andsb≠srs\_\{b\}\\neq s\_\{r\}\. We randomizesas\_\{a\}andsbs\_\{b\}to limit the effects of positional bias, and the resulting generated triplets for specific pools are kept consistent across experiments\. In total,\|S\|×\(\|G\|2\)\|S\|\\times\{\|G\|\\choose 2\}triplets are selected in a particular pool\.

## Appendix DHuman Evaluation

In this section, we provide additional details on the human evaluation setup\.

### D\.1Sampling Triplets

We utilize a stratified sampling procedure to generate the human evaluation triplets\. As we are specifically interested in comparisons that include human\-written stories, our sampling strategy explicitly upsamples those triplets\. To construct a triplet, we first randomly select a writing prompt and perform a weighted sampling for a reference generatorrr, assigning a weight of33to humans and11to all others\. We then sample candidate generatorsaaandbb, ensuring that at least30%30\\%of the time eitheraaorbbis human; for the remaining70%70\\%of cases,aaandbbare sampled uniformly at random without replacement\. Finally, one of the two stories per prompt per generator is uniformly sampled, yielding\(sr,sa,sb\)\(s\_\{r\},s\_\{a\},s\_\{b\}\)\. We note that one of the writing prompts \(and all stories corresponding to it\) was excluded from this process, as it was used for an example in the guidelines given to the annotators\.

### D\.2Annotation Setup

Annotations were collected via an online annotation platform \(omitted for anonymity\)\. To account for potential cultural or linguistic biases in narrative interpretation, we sourced annotators from two distinct geographic cohorts: India and Nigeria\. Both regions possess a large population of fluent English speakers but offer diverse cultural lenses through which narrative tropes and similarities might be perceived\.

As mentioned in[Section˜4\.1](https://arxiv.org/html/2606.17350#S4.SS1), we remove all metadata and paratext, including any identifying information about the authors, and only retain the text of the prompts and stories\. Ther/WritingPromptssubreddit rules444https://www\.reddit\.com/r/WritingPrompts/wiki/rulesexplicitly forbid harmful content, and we additionally conduct an automated check with Gemini 3\.1 Pro across both the human and model\-generated stories\.

Every triplet was annotated once by both cohorts, resulting in two independent human judgments per triplet\. The expert guidelines used are provided in full in[Figure˜12](https://arxiv.org/html/2606.17350#A8.F12)and[Figure˜13](https://arxiv.org/html/2606.17350#A8.F13)\. Annotators were compensated at an hourly rate of $19\.5 \($1\.3 per triplet with an estimated annotation time of 4 minutes per triplet\), well above the median wage in both regions\(International Labour Organization,[2024](https://arxiv.org/html/2606.17350#bib.bib19)\)\.

### D\.3Triplet Categories

In aggregate, the exact models utilized often matters less than the nature of the comparison—for instance, we care more about whether a human reference is being compared to two model\-generated candidates, or whether a model reference is being compared to another generation from the same model and a generation from a different model, than we do about which specific models are being compared\. Thus, when reporting certain statistics such as annotator agreement, we abstract the specific generator identities within each triplet into broader categories\.

Specifically, for a given triplet, each corresponding generator is converted to eitherH, denoting human, orMk\(wherek∈\{1,2,3\}\\in\\\{1,2,3\\\}\), denoting a model\. The subscriptkserves to distinguish distinct models within the triplet\. The order of the candidate stories do not matter; for example,\(M1, H, M2\)and\(M1, M2, H\)are both categorized the same, and denoted as the former\.

To illustrate, the category\(H, M1, M2\)isolates all instances where a human reference is compared against two different models\. Similarly, the category\(M1, M1, M2\)isolates self\-similarity comparisons, where a model reference is compared to another generation from the same model, and a generation from a different model\. By partitioning the data in this manner, we can independently assess statistics across structurally distinct categories of the task\.

### D\.4Human and LLM Judge Confidences

![Refer to caption](https://arxiv.org/html/2606.17350v1/x6.png)Figure 6:Average LLM judge agreement with human annotators split by reported confidence\. Neither the LLM judge nor the human annotators ever reported a confidence score of 1\.As both human annotators and the LLM judge provide confidence scores, we compile a heatmap showcasing average agreement across confidence scores in[Figure˜6](https://arxiv.org/html/2606.17350#A4.F6)\. We note that while the confidence scale ranged from 1 to 5, neither the LLM judge nor the human annotators ever provided a confidence score of 1, corresponding to the case where both stories are equally similar to the reference\. For the cases where the LLM judge chose a confidence score of 2\-3, the agreement is essentially at random regardless of annotator confidence\. Interestingly, when the LLM judge selects a confidence score of 4, agreement is moderate when human annotator confidence is low \(2\-3\), but reverts to random chance when human annotator confidence is high \(4\-5\)\. Nonetheless, for the majority of samples, the LLM judge selects a confidence score of 5\. In these cases, agreement is particularly high when human annotators are also confident, with94\.6%94\.6\\%agreement with human annotator confidence44and97\.0%97\.0\\%with human annotator confidence55\. In the rarer cases where human annotators have low confidence, agreement is low but above random chance, at66\.7%66\.7\\%for a human annotator confidence of11\. Overall, the LLM judge agrees with annotators92\.8%92\.8\\%when it selects a confidence score of 5, which it does for6969of the triplets\. This suggests that a confidence cutoff of55can be used to improve the reliability of the LLM judge annotations, though it is likely that will bias the resulting annotation set towards the easier triplets\.

## Appendix EAutomated Annotation

![Refer to caption](https://arxiv.org/html/2606.17350v1/x7.png)Figure 7:Similarity heatmap of triplets annotated by the LLM judge, showcasing the normalized selection rate within the closed\-source, open\-source, and OLMo generator pools without a confidence filter\. The rows indicate the reference and the columns indicate the selected candidate generator, as detailed in[Section˜3\.1](https://arxiv.org/html/2606.17350#S3.SS1)\.![Refer to caption](https://arxiv.org/html/2606.17350v1/x8.png)Figure 8:Similarity heatmap of triplets annotated by the preference model, within the closed source, open source, and OLMo pools\.![Refer to caption](https://arxiv.org/html/2606.17350v1/x9.png)Figure 9:Similarity heatmap of triplets annotated by the LLM judge, showcasing the normalized selection rate within the closed\-source, open\-source, and OLMo generator pools under therandomsetting \(all three stories in a triplet based on different writing prompts\)\. The rows indicate the reference and the columns indicate the selected candidate generator, as detailed in[Section˜3\.1](https://arxiv.org/html/2606.17350#S3.SS1)\.We provide additional details for all three automated annotation frameworks in this section\.

### E\.1LLM\-as\-a\-Judge

The LLM\-as\-a\-Judge setup directly utilizes an LLM to judge the similarity given a triplet\. Inspired by prior approaches to narrative evaluation, we decompose narrative similarity into several simple categories such as plot, themes, and characters\(Paech,[2023](https://arxiv.org/html/2606.17350#bib.bib36); Hatzelet al\.,[2026](https://arxiv.org/html/2606.17350#bib.bib17)\)\. GPT\-5 \(gpt\-5\) is used as the judge model for all our experiments\. The full prompt used for judging is given in[Figure˜14](https://arxiv.org/html/2606.17350#A8.F14)\.

### E\.2Narrative Component Embedding

For narrative component embedding, narrative components are extracted by an LLM from each story, and they are transformed into embeddings\. Cosine similarity between the embeddings are then used to annotate the triplets\. We use GPT\-5 to extract the components, utilizing the framework opted by the top\-scoring system inHatzelet al\.\([2026](https://arxiv.org/html/2606.17350#bib.bib17)\), where each story is decomposed into overall abstract theme, course of action, and outcomes\. We then use the Gemini Embedding \(gemini\-embedding\-001\) to embed all three components together as a concatenated string\. The full prompt used for extracting narrative components is given in[Figure˜15](https://arxiv.org/html/2606.17350#A8.F15)\.

### E\.3Preference Model

Table 3:Hyperparameters used for training the preference model\.We utilize a discriminative Bradley\-Terry \(BT\) formulation\(Bradley and Terry,[1952](https://arxiv.org/html/2606.17350#bib.bib5)\), where in each annotated triplet\(sr,ss,sd\)\(s\_\{r\},s\_\{s\},s\_\{d\}\), the preference modelrθr\_\{\\theta\}learns to assign a higher scalar value to the more similar candidatesss\_\{s\}than to the less similar candidatesds\_\{d\}when evaluated against the referencesrs\_\{r\}\. The corresponding loss is defined as:

ℒ=−log⁡\(σ​\(rθ​\(sr,ss\)−rθ​\(sr,sd\)\)\)\\mathcal\{L\}=\-\\log\\left\(\\sigma\\left\(r\_\{\\theta\}\(s\_\{r\},s\_\{s\}\)\-r\_\{\\theta\}\(s\_\{r\},s\_\{d\}\)\\right\)\\right\)
We use Qwen3 1\.7B \(Qwen/Qwen3\-1\.7B\) as our base model, appending a linear layer at the end and conducting full fine\-tuning with the aforementioned formulation\. We aggregate the GPT\-5 annotations across the closed\-source, open\-source, and OLMo pools to create the training dataset\. As we are using the human\-annotated triplets for evaluation, we exclude all triplets with any overlap in stories\. We reserve2%2\\%of the preference pairs to serve as an evaluation set and utilizing the remainder for training\. We use AdamW as the optimizer\. The hyperparameters used for training are provided in[Table˜3](https://arxiv.org/html/2606.17350#A5.T3)\. The model was trained on a single NVIDIA RTX A6000 48GB GPU, taking approximately 40 minutes to train\.

## Appendix FAdditional Results

In this section, we provide additional results of the experiments described in the main text\.

### F\.1Within Model Categories, Without Confidence Filter

[Figure˜7](https://arxiv.org/html/2606.17350#A5.F7)provide the resulting heatmaps from the comparison done in[Section˜6\.2](https://arxiv.org/html/2606.17350#S6.SS2), but without the confidence filter applied\. We note that all trends we point out in[Section˜6\.2](https://arxiv.org/html/2606.17350#S6.SS2)persist\.

### F\.2Preference Model Heatmaps

[Figure˜8](https://arxiv.org/html/2606.17350#A5.F8)provides the resulting heatmaps of the setting described in[Section˜6\.2](https://arxiv.org/html/2606.17350#S6.SS2), but annotated by the preference model instead of the LLM judge\. We note that the overall trends remain, and the results are largely consistent with the LLM judge annotations, providing further evidence for its reliability\.

### F\.3Sampling Parameter Sweep Heatmaps

[Figure˜11](https://arxiv.org/html/2606.17350#A8.F11)provides the full heatmaps of the sampling parameter sweep described in[Section˜7\.2](https://arxiv.org/html/2606.17350#S7.SS2)\. As we detailed, we find that top\-P has no discernible effect on the resulting diversity, while temperature only has a notable effect on the OLMo SFT model\.

## Appendix GAdditional Experiments

In this section, we describe additional experiments that were omitted from the main text for brevity\.

### G\.1Preference Model Evaluation with Narrative Component Extraction

As we directly train on LLM\-annotated triplets, it is plausible that the preference model’s scores may be based on other factors of similarity, such as lexical and syntactic distribution, which can coincide with narrative similarity in our dataset\. To test this, we also conduct a separate evaluation where we remove all stylistic properties from the stories by extracting narrative components via GPT\-5\. We use the same prompt as narrative component embedding, extracting the story into overall abstract theme, course of action, and outcomes\.

Table 4:Average preference model agreement with human annotations across different triplet categories, using the original stories \(PM\) as well as the agreement when using extracted narrative components \(PM Comp\.\)\.We compare the preference model annotations against the human annotations, and compile the average agreement in[Table˜4](https://arxiv.org/html/2606.17350#A7.T4)\. Both the component\-extracted and original story evaluations report similar results, despite the component\-extracted format not being included in the training data\. This indicates that the preference model is robust to surface\-level linguistic variations, and can likely be used as a general scoring model for similarity assessment\.

### G\.2Random Comparison

In addition to the prompt\-wise comparisons in[Section˜6](https://arxiv.org/html/2606.17350#S6), we also test the case where all three stories are from different writing prompts\. Like in[Section˜4\.3](https://arxiv.org/html/2606.17350#S4.SS3), we iterate through all storiessr∈Ss\_\{r\}\\in S, with corresponding promptprp\_\{r\}, and combinations of generators\{a,b\}∈\(G2\)\\\{a,b\\\}\\in\{G\\choose 2\}\. However, we then randomly sample from all stories generated byaaandbbrespectively, such that all three are from mutually distinct prompts\. As all three stories are written for different prompts, we hypothesize that the task becomes more ambiguous, and additional noise is introduced in the form of prompt selection\. For example, in a triplet where the reference story prompt is distinctly more similar to one of the candidate story prompts, the narrative structure will likely follow suit regardless of the generators\. Thus, we consider this setting to be less useful in making inferences, but still provide it for comparison\.

We apply this setting for the same pools and annotation schemes as[Section˜6\.2](https://arxiv.org/html/2606.17350#S6.SS2), with the resulting heatmaps compiled in[Figure˜9](https://arxiv.org/html/2606.17350#A5.F9)\. While the overall trends we pointed out in[Section˜6\.2](https://arxiv.org/html/2606.17350#S6.SS2)still remain, the magnitude is less pronounced, with LLM stories being judged to be more similar to a human story up to a31\.9%31\.9\\%of the time\. As we detailed above, this setting is inherently more ambiguous with additional noise, and so the reduction in magnitude is to be expected\. Nevertheless, even with completely different prompts, human\-written stories are distinctly less similar to the LLM\-generated ones\.

### G\.3Chain\-of\-Thought

![Refer to caption](https://arxiv.org/html/2606.17350v1/x10.png)Figure 10:Similarity heatmap of triplets annotated by the preference model, showcasing the normalized selection rate comparing thinking and non\-thinking variants of Qwen3 and OLMo3\.1\. The rows indicate the reference and the columns indicate the selected candidate generator, as detailed in[Section˜3\.1](https://arxiv.org/html/2606.17350#S3.SS1)\.We test whether chain\-of\-thought reasoning has any effect on narrative diversity by comparing the regular and chain\-of\-thought \(or thinking\) variants of Qwen3 and OLMo 3\.1\. For Qwen3 we simply enable and disable chain\-of\-thought via the system prompt, while for OLMo 3\.1 we compare the OLMo 3\.1 Thinking and OLMo 3\.1 Instruct models\. The results are compiled in[Figure˜10](https://arxiv.org/html/2606.17350#A7.F10)\. We note that with thinking enabled, Qwen3 has a higher self\-similarity at89\.889\.8vs83\.583\.5, but the OLMo Thinking variant has a lower self\-similarity at93\.593\.5vs89\.889\.8\. In general, we find little meaningful differences overall, and so we conclude that chain\-of\-thought prompting may at best mildly affect the diversity, specific to the model family\.

## Appendix HAI Use

We utilize AI coding assistants \(available via GitHub Copilot\) in some cases to generate boilerplate code for experiments and data visualization\. All generated code was reviewed and tested by the authors\. We also use Gemini 3\.1 Pro as a general\-purpose assistant for formatting LaTeX elements and for checking grammar and typographical errors in the final drafts of the manuscript\. No AI system was used to generate novel ideas or draw analytical conclusions, and no part of the manuscript was generated by such a system\.

![Refer to caption](https://arxiv.org/html/2606.17350v1/x11.png)Figure 11:Similarity heatmap of triplets annotated by the preference model, within the OLMo pool under a sweep of sampling parameters, with all combinations of temperatureT∈\{0\.2,0\.4,0\.6,0\.8,1\.0\}T\\in\\\{0\.2,0\.4,0\.6,0\.8,1\.0\\\}and top\-PP∈\{0\.85,0\.9,0\.95,1\.0\}P\\in\\\{0\.85,0\.9,0\.95,1\.0\\\}\.Narrative Similarity Annotation Guidelines \(1 of 2\)Task OverviewIn this study, you are tasked with identifying similar stories\. In each sample, three short stories of roughly 200\-400 words will be provided: a reference story, story A, and story B\. You will compare two candidate stories \(story A and B\) against the reference story and determine which candidate is more narratively similar to the reference\. You will also be asked to rate your confidence in your judgement\. All stories are written responses to the same writing prompt\. Your annotations will be used for academic research\.Task Workflow1\.Read theReference Storycarefully, noting its key narrative elements\.2\.ReadStory AandStory Bcompletely\.3\.Compare each candidate’s narrative structure to the reference\.4\.Select which story \(A or B\) is more narratively similar\.5\.Rate your confidence \(1\-5\) in this choice\.Narrative SimilarityThe narrative similarity of stories can be broken down into three core aspects: \(1\) the abstract themes of the story, \(2\) the course of action, and \(3\) the story outcomes\. At one extreme, this means that the story deals with the same themes and tells the same order of events with an identical outcome or conclusion, just using a different wording; at the other extreme, the story might be completely different and lack any basis for comparison\.More difficult to assess are stories that only share some similarities\. In such cases, you are asked to weigh the three core components of story similarity\. You should focus on the core aspects of stories, potentially largely ignoring side storylines\. How you weigh the individual factors should be based on your intuitive impression of which aspects you consider crucial to the overall similarity of the specific stories\.We define these three aspects as follows:•Abstract Themedescribes the defining constellation of problems, central ideas, and core motifs of a story\. The definition does not cover the concrete setting of a story\.•Course of Actiondescribes sequences of events, actions, conflicts, and turning points in a story and the order in which they happen\.•Outcomesdescribe the results of the plot at the end of the text, for example, the conflict resolution, the characters’ fates, moral lessons, etc\. It does not cover intermediate statuses that change later in the story\.Each aspect can take different forms in an actual pair of stories\. Below, we list one example for each aspect:•The general setting of the story, if it strongly influences the events in the story or the events necessitate a specific setting \(abstract themes\)–A: On the week\-long journey from Europe to the Americas, the crew members get into a heated conflict about the best ration packages\.–B: The flight to Mars is long\. After several weeks, the astronauts become better friends than ever before, having to share the limited resources\.–A and B share some similarities in that the polar opposite outcomes are both enabled by being cut off from the outside world\.•The order of events in the story \(course of action\)–A: After the ship capsizes and Alice barely makes it out alive, she starts living life to the fullest\.–B: Alice is living life to the fullest until, one day, her ship capsizes\. She barely makes it out alive\.–A and B are similar in that both tell of a good life and a shipwreck \(abstract theme\), but they differ in the course of action, and the order is very different\.•The outcomes of events \(story outcomes\)–A: The man intentionally drops a cup; it breaks\.–B: He accidentally swipes the bottle off the table, and it shatters\.–A and B are similar in that the events are comparable and lead to similar outcomes\.Figure 12:The annotation guidelines provided to human annotators \(part 1 of 2\)\.Narrative Similarity Annotation Guidelines \(2 of 2\)There are a range of factors that expressly do NOT contribute to the narrative similarity:•The style of writing in a story•The concrete setting of a story \(also including the time period\)\.•The names of the characters and locations•The length of a text•The level of detail in which the events are toldDifferentiating Between Similarity AspectsDistinguishing the three aspects can be challenging\. In general, it is important to consider each aspect independently\.Often, pairs of stories that are similar in terms of course of action will also share an abstract theme\. However, it is possible that similar events emerge from completely different surrounding circumstances\. Outcomes, on the other hand, are clearly distinct from the other two aspects: practically identical events in stories with comparable abstract themes can result in polar opposite outcomes\.When comparing abstract themes, it can help to explicitly formulate them\. There is, of course, no single correct answer, and a single story’s theme could be formulated in many ways\. Two stories share a general theme if there is a description that captures the defining circumstances of both stories\.Confidence ScaleWhen selecting the confidence, it is important to note that it should not necessarily be the amount of similarity between the reference and chosen candidate story, but rather your confidence that the chosen candidate story is more similar to the reference than the other candidate story\. In specific, each score can be interpreted as written below:•1: No confidence; Both stories A and B are equally similar to the reference story in narrative•2: Slight confidence; The similarities are comparable, and it could go either way•3: Moderate confidence; One story is noticeably more similar to the reference, but there is room for doubt•4\-5: Strong confidence; One story is clearly and distinctly more similar to the referenceExampleIn the following example, summaries are used for faster reading, but the task itself will use the full short stories\.Reference: Mara writes a poignant, unsent letter to her deceased brother, Noah, reminiscing about their shared childhood and updating him on her life\. She ultimately places the letter in a drawer filled with dozens of others, illustrating her quiet, ongoing process of grieving and holding onto his memory\.Story A: Margaret writes an anguished letter to Steve, longing for the “faithful, loyal man” she once had and expressing deep guilt over trying to replace him with someone who ultimately broke her trust\. It is only in the final sign\-off that the true nature of their relationship is revealed; the writer is a daughter mourning the father she lost nineteen years ago\.Story B: A young man drafts a nostalgic letter to her friend, Sarah, recalling their vibrant past and the small, beautiful details of their shared life\. Even though Sarah is in a facility suffering from severe dementia and may not understand the words, he mails the letter anyway, deciding that love does not require memory to exist\.Choice: Story B, Confidence 5All three stories are nostalgic letters reflecting on the shared history of a lost loved one\. Story A diverges from the reference by using misdirection to imply that the letter is initially to a lover, and includes an exploration of her guilt over “trying to replace him”\. While Story B diverges in some surface details \(young man writing instead of a named woman, writing to a friend instead of family\), it shares the reference’s straightforward focus on a deeply rooted childhood bond\. Thus, Story B is more narratively similar\.Figure 13:The annotation guidelines provided to human annotators \(part 2 of 2\)\.Prompt for LLM\-as\-a\-Judge triplet annotation\.You are an expert literary analyst\. Your task is to critically evaluate narrative similarity between two candidate stories and a reference story\.You will be given:\- A reference story\- Two candidate stories \(Writer A & Writer B\)Your job is to determine which candidate story is more similar to the reference in several narrative criteria\.\[REFERENCE STORY\]\{story\_ref\}\[/REFERENCE STORY\] \[WRITER A\]\{story\_a\}\[/WRITER A\] \[WRITER B\]\{story\_b\}\[/WRITER B\]Judge which writer’s story more closely matches the reference story on each of these dimensions:\- Plot similarity \(events, structure, narrative arc\)\- Character similarity \(roles, motivations, arcs, emotional dynamics\)\- Setting similarity \(time, place, atmosphere\)\- Tone & mood similarity \(emotional feel, narrative voice\)\- Theme similarity \(underlying message, core concepts\)\- Style similarity \(pacing, structure, point of view\)\- Overall similarity \(general closeness to reference story\)Judging notes:\- Surface details \(e\.g\., character names, specific locations, diction\) do not matter; focus on the narrative elements\.\- Be aware that these similarities may be independent, i\.e\. a story may be more similar to the reference in one and less in another\.\- Outputs will sometimes be truncated to ensure length consistency\. Don’t penalise this, just judge what is there on its merit\.\- You must always pick a winner for each criteria \(no draws\)For each criterion, assign a winner \(Writer A or Writer B\), like so:```
"plot_similarity": "A"
"character_similarity": "B"
"setting_similarity": "A"
```

In addition to picking a winner, you should also assign a confidence rating for the overall similarity, which should reflect the confidence in disambiguation\. The score should be between 1 \(both stories extremely similar, too similar to consistently tell apart\) to 5 \(the winning story is explicitly and concretely more similar\)\.Respond in valid json without additional commentary \(remembering to escape any string quotes\), in this format:```
{
"analysis": "detailed analysis and reasoning about the coming scoring decisions",
"plot_similarity": "winner & disparity rating",
"character_similarity": "winner & disparity rating",
"setting_similarity": "winner & disparity rating",
"tone_mood_similarity": "winner & disparity rating",
"theme_similarity": "winner & disparity rating",
"style_similarity": "winner & disparity rating",
"overall_similarity": "winner & disparity rating",
"confidence": "confidence rating for overall similarity"
}
```

Figure 14:Prompt for LLM\-as\-a\-Judge triplet annotation\.Prompt for Narrative Component ExtractionDescribe a given story in JSON format\. You need to describe the following three components:1\) Overall abstract theme: Describe in brief the central ideas, core motifs and defining constellation of problems\. For example, in both these stories:A: "On the week\-long journey from Europe to the Americas, the crew members get into a heated conflict about the best ration packages\."B: "The flight to Mars is long\. After several weeks, the astronauts become better friends than ever before, having to share the limited resources\." Theme: A story about people isolated from outside world in a journey, and how it affects their interpersonal relationship\.2\) Course of action/events: Describe in brief the sequence of events that actually happens in the story\. For example, in the following stories:A: "After the ship capsizes and Alice barely makes it out alive, she starts living life to the fullest with a new\-found perspective about how precious life is\." Events: Alice’s ship capsizes\. Alice barely makes it out alive\. Alice starts living life to the fullest\.B: "Alex loses his engagement ring while swimming\. He freaks out, and after hours of diving for it, he still cannot find it\." Events: Alex loses his engagement ring while swimming\. Alex freaks out\. Alex looks for it\. Alex fails to find it\.3\) The outcomes: Describe in brief the final ending or outcomes of the story\. For example, in the following stories:A: "Anna loses her purse\. She retraces her steps but cannot find it\. Dan finds it and helpfully returns it to her\." Outcome: Someone finds a lost item and returns to owner\.B: "Brian lost his backpack\. He was terrified because there were important documents in it\. After an hour of intense search he finally found it\." Outcome: Someone finds their lost item\.C: "Jill was driving home when another car suddenly crashed into hers\. After receiving medical attention, she recovered within just days and now advocates for traffic safety\."Outcome: A person advocates for traffic safety after recovering from car crash\.You should produce a valid JSON object with the three attributes describing the given story: "theme", "events" and "outcome"\. Do not produce any extra explanation or additional text\.Figure 15:Prompt for Narrative Component Extraction\.Prompts for Sequential GenerationSystem PromptYou are a creative writer tasked with writing stories based on given prompts and word count requirements\. Do not reply with anything other than the story itself\. Do not include any commentary, explanations, or notes, just the story\.User PromptWrite a short story between \{min\_words\} and \{max\_words\} words based on the writing prompt: \{wp\}Follow\-up User Prompt for Sequential Negative GenerationGenerate another story based on the same writing prompt and word range\. Try to make it as different from previous stories as possible\.Figure 16:All LLM prompts used for generating stories\.