GENIE: A Fine-Grained Measure for Novelty

# Genie: A Fine-Grained Measure for Novelty
Source: [https://arxiv.org/html/2606.12790](https://arxiv.org/html/2606.12790)
Ramya Namuduri1Manya Wadhwa2Anshun Asher Zheng1 Greg Durrett2Junyi Jessy Li1 1The University of Texas at Austin2New York University ramya\.namuduri@utexas\.edu

###### Abstract

Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks\. Prior work has focused on addressingwhethermodels are capable of generating creative outputs\. Here, we aim to considernoveltyand investigate what makes model\-generated content novel or not novel in a task\-specific manner\. We propose a fine\-grained evaluation metricGenieto measure the novelty of responses along task\-specificfeatureswith respect to a population of responses\. We show that unlikeGenie, holistic metrics struggle to capture the high\-dimensionality of novelty and do not provide insight on which properties they target\. Finally, we useGenieto measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty\.

![[Uncaptioned image]](https://arxiv.org/html/2606.12790v1/figures/genie.png)Genie: A Fine\-Grained Measure for Novelty

Ramya Namuduri1Manya Wadhwa2Anshun Asher Zheng1Greg Durrett2Junyi Jessy Li11The University of Texas at Austin2New York Universityramya\.namuduri@utexas\.edu

[Code](https://github.com/AlliteraryAlligator/GENIE/tree/main)[![[Uncaptioned image]](https://arxiv.org/html/2606.12790v1/figures/hf-logo.png)Data](https://huggingface.co/collections/AlliteraryAlligator/genie)[Project Page](https://alliteraryalligator.github.io/GENIE/)

## 1Introduction

Can LLMs produce creative work? Prior work has discovered that models struggle to generate diverse and creative outputs given various tasksJiang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib19)\), using lexical\-semantic approachesZhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\); Lu et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib26)\); Ismayilzada et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib18)\), creativity testsWenger and Kenett \([2025](https://arxiv.org/html/2606.12790#bib.bib49)\); Wadhwa et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib46)\)and human expertsChakrabarty et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib6)\)\. Model creativity has been tested on a variety of tasks using a multitude of metrics\. However, past work has not isolated preciselywhere models lack creativityon specific tasks\.

Novelty is a key component of creativityBoden \([1991](https://arxiv.org/html/2606.12790#bib.bib3)\), where the uniqueness of a response can be attributed to different reasons specific to the task\. Features such asplotandcharacter developmentare relevant to narrative writing, whilemotivationandmethodologyare applicable to scientific discovery\. Two responses can be strikingly original, and yet differ vastly in what makes them novel\. Similarly, a response might appear novel in some ways and commonplace in others\. For instance, Figure[1](https://arxiv.org/html/2606.12790#S1.F1)illustrates how an LLM\-generated response can concurrently have unique and mundane features\. However, this fine\-grained novelty is not captured by many existing holistic creativity metrics\. Cosine distance against other responses, and other creativity metrics\(Zhang et al\.,[2025](https://arxiv.org/html/2606.12790#bib.bib55); Chakrabarty et al\.,[2025a](https://arxiv.org/html/2606.12790#bib.bib7); Fein et al\.,[2026](https://arxiv.org/html/2606.12790#bib.bib11); Shaib et al\.,[2024](https://arxiv.org/html/2606.12790#bib.bib42)\), although informative, do not describe why an instance is more or less novel\.

![Refer to caption](https://arxiv.org/html/2606.12790v1/x1.png)Figure 1:High level motivation forGenie\. Existing holistic metrics do not capture the high\-dimensionality of novelty whileGenie’s fine\-grained approach provides explainability\.We proposeGenie\(GranularEvaluation ofNovelIdeas withExplainability\), a framework that quantifies the fine\-grained nuances that differentiate equally novel responses by extractingfeaturespresent in the responses, with respect to a pre\-defined population\. Inspired by alternative semantics of questions literature\(Hamblin,[1957](https://arxiv.org/html/2606.12790#bib.bib14); Karttunen,[1977](https://arxiv.org/html/2606.12790#bib.bib21); Lahiri,[2001](https://arxiv.org/html/2606.12790#bib.bib24)\), we do so by generating and mapping questions to automatically discovered features relevant to the task, and measuring fine\-grained novelty through the similarities of their answers\.

Genie’s focus on automatically extractedfeaturesisolates novelty from other components of creativity, such as value \(i\.e\., quality;Boden \([1991](https://arxiv.org/html/2606.12790#bib.bib3)\)\)\. This is in contrast to some existing metrics that subsume the quality of responsesChakrabarty et al\. \([2025a](https://arxiv.org/html/2606.12790#bib.bib7)\)or the predicted aesthetic valueFein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\)\. By tethering the feature extraction to thetask,Genie’s approach is inherently domain agnostic\.

Genie’s level of abstraction \(using questions and answers\) allows it to treat models of varying instruction\-following capabilities comparably, specifically with regards to models with poor instruction\-following capabilities that exhibit greater diversity despite lower qualityWest and Potts \([2025](https://arxiv.org/html/2606.12790#bib.bib50)\); Le Bronnec et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib25)\); Hamilton \([2024](https://arxiv.org/html/2606.12790#bib.bib15)\)\.

Furthermore, determining the uniqueness of responses holistically without a reference point can be subjective\. Prior work has used the pretraining corpusLu et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib26)\), other model\-generatedJiang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib19)\), humanWenger and Kenett \([2025](https://arxiv.org/html/2606.12790#bib.bib49)\)or self\-generated\(Zhang et al\.,[2026](https://arxiv.org/html/2606.12790#bib.bib54),[2025](https://arxiv.org/html/2606.12790#bib.bib55)\)responses as reference points for their measurements\. In this work, we make the references explicit:Geniemeasures noveltywith respect to a population; this population is defined by the user\.

We explore the creative writing domain\.Genieindependently found77fine\-grained features pertaining to this task, which roughly correspond to theoretically defined taxonomies\(Hamilton et al\.,[2026](https://arxiv.org/html/2606.12790#bib.bib16)\)\. We constructed and used a population of responses generated by 21 LLMs to evaluate thesensitivityofGenieon pairs of LLM\-generated responses that minimally differ along these features\.Geniesuccessfully detects differences in fine\-grained novelty across such pairs\.Geniealso demonstratesrobustnesson pairs that only differ via paraphrasing; it does*not*detect a significant difference along any feature in these cases\. Additionally, we performed human studies to intrinsically evaluate various components that compriseGenie\.

Genie’s fine\-grained nature provides a diagnostic lens into existing holistic metrics used to measure novelty, diversity and creativity\. We found that they either capture extraneous properties that sometimes spuriously correlate with novelty, or are lexically sensitive\. Furthermore, many existing mitigation methods, such as alternate promptingZhang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib54)\); Huot et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib17)\)and decoding strategiesPeeperkorn et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib35)\)for improving creativity have been evaluated on these holistic metrics\. We demonstrate howGenieprovides a more explainable evaluation, shedding light into*which*features, if any, these methods actually improve\.

![Refer to caption](https://arxiv.org/html/2606.12790v1/x2.png)Figure 2:Geniecaptures fine\-grained novelty of a target response with respect to a population of responses using automatically discovered task\-specific features\. Here,Geniediscovers features and their descriptions for creative writing\. We generate questions for a given promptpp\(Step3\) and map each question to a feature\. Step 2 computes answers to questions from the population\. Finally,Geniemeasures the novelty of a target response with respect to this population, along each feature \(e\.g\. setting or plot\) using the average pairwise answer dissimilarity\.
## 2Related Work

A growing line of work examines whether LLMs are creative, drawing on theoretical frameworks from cognitive science and computational creativity\. Many studies operationalize Boden’s theories of creativityFranceschelli and Musolesi \([2024](https://arxiv.org/html/2606.12790#bib.bib12)\); Ismayilzada et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib18)\); Schapiro et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib41)\); Nagarajan et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib31)\); Wadhwa et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib46)\)while others adapt psychometric tests such as Torrance Tests of Creative ThinkingChakrabarty et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib6)\)\. Past work has also focused on the homogenizing effects of LLMs when looking at a population of LLM outputsWenger and Kenett \([2025](https://arxiv.org/html/2606.12790#bib.bib49)\); Jiang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib19)\)and attributed this to post\-training alignment methodsPadmakumar and He \([2024](https://arxiv.org/html/2606.12790#bib.bib33)\); West and Potts \([2025](https://arxiv.org/html/2606.12790#bib.bib50)\); Slocum et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib43)\)\. There are also domain specific benchmarks that evaluate creativity in writingFein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\); Chakrabarty et al\. \([2025a](https://arxiv.org/html/2606.12790#bib.bib7)\); Xu et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib51)\); Fein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\); Zhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\)\. Most of this work asks whether models are creative; instead we ask where a given response is novel and decompose this judgment along task\-specific features\.

Recent work has proposed explicit metrics and benchmarks for evaluating novelty and creativity in text generation\. One common approach is to use the pretraining corpus as reference to quantify originality as a fraction of n\-gramsLu et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib26)\); Padmakumar et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib34)\); Merrill et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib28)\)\. NoveltyBenchZhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\)operationalizes diversity by checking for if an output belongs to one of the preexisting clusters of unique responses\. Past work also trains reward models on human preference data to score creative writing directlyChakrabarty et al\. \([2025a](https://arxiv.org/html/2606.12790#bib.bib7)\); Fein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\); Cao et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib5)\)\. Across all the prior work, novelty is reduced to a scalar where the reference is implicit or calculated with surface units \(e\.g\., n\-grams or distributional semantics\)\.QUDsimNamuduri et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib32)\)decomposes similarity along discourse\-level questions under discussion \(QUDs\)\(Roberts,[2012](https://arxiv.org/html/2606.12790#bib.bib39); Velleman and Beaver,[2016](https://arxiv.org/html/2606.12790#bib.bib45); Beaver et al\.,[2017](https://arxiv.org/html/2606.12790#bib.bib2)\)\. This established the effectiveness of measuring the similarity of answers to the same questions across documents\. However, QUDs capture discourse, whereasGeniedecomposes novelty into task\-level features against an explicit population, separating content from quality and aesthetics of the output being evaluated\.

## 3TheGenieMeasure of Novelty

Consider a population of model\-generated documents𝒟p​o​p=\{d1,…,dm\}\\mathcal\{D\}\_\{pop\}=\\\{d\_\{1\},\.\.\.,d\_\{m\}\\\}generated from promptPP\. The novelty of a target documentdt∉𝒟p​o​pd\_\{t\}\\notin\\mathcal\{D\}\_\{pop\}is defined as its relative dissimilarityddfor alld∈𝒟p​o​pd\\in\\mathcal\{D\}\_\{pop\}along variousfeatures\. Intuitively, unique responses share fewer commonalities across features with other responses\.Genieis explainable, because each feature is associated with natural language definitions; when unpacked, further explainability can be obtained by pinpointing prompt\-specific questions under that feature, as well as their natural language answers whose similarity we calculate\.

### 3\.1Features

Features are task\-specific axes that aim to maximize coverage over different characteristics of the task\. For instance, a task like creative writing might include features such asplot,setting, andperspective\. On the other hand,motivationandmethodologymight be relevant to evaluating the novelty of scientific discovery\. Manually defining these features for tasks can be tedious, subjective, or infeasible \(due to lack of expertise\)\. Instead, we automaticallydiscoverfeatures\.

We prompt an LLM to form features with descriptions given a task \(e\.g\. creative writing\)kktimes\. The resulting set is clustered to find commonly recurring features\. Concretely, we discoverNNclusters and label them asℱ=\{f1,…,fN\}\\mathcal\{F\}=\\\{f\_\{1\},\.\.\.,f\_\{N\}\\\}to form the set offeatures\. Step1and2in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)demonstrate how we discover features for the creative writing task\. Each featureffis a functionf:𝒟→𝒱ff:\\mathcal\{D\}\\to\\mathcal\{V\}\_\{f\}that maps a document to its feature value, where𝒟\\mathcal\{D\}the set of all documents and𝒱f\\mathcal\{V\}\_\{f\}is the set of all possible values the feature can take\.

### 3\.2GenieScoring

Genieelicits values for each featuref∈ℱf\\in\\mathcal\{F\}in a document \(i\.e\.,f​\(d\)f\(d\)\) via question generation and answering\. Here, question generation is an intermediate process that groundsℱ\\mathcal\{F\}to prompt\-specific dimensions, allowing feature values to be more reliably extracted by answering the questions\. This paradigm takes the alternative semantics view of questions\(Hamblin,[1957](https://arxiv.org/html/2606.12790#bib.bib14); Karttunen,[1977](https://arxiv.org/html/2606.12790#bib.bib21); Lahiri,[2001](https://arxiv.org/html/2606.12790#bib.bib24)\), where hierarchically organized questions are used to partition the possible worlds into subspaces that “answer the same questions”, in our case, “values of the same featureff”, hence allowing us to measure novelty*for each feature*\.

##### Prompt\-Specific Decomposition

Given a promptp∈Pp\\in P, we generate a set of questions𝒬=\{q1,…,qj\}\\mathcal\{Q\}=\\\{q\_\{1\},\.\.\.,q\_\{j\}\\\}\(see Step3in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)\)\. The set𝒬\\mathcal\{Q\}is formed without knowledge ofℱ\\mathcal\{F\}so that the resulting questions are specific topp, while not overfitting to the taxonomy of discovered features\.

##### Feature Mapping

We then map each question in𝒬\\mathcal\{Q\}to exactly onefeaturef∈ℱf\\in\\mathcal\{F\}\(Step3in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)\)\. The subset of questions that map to featureffis denoted by𝒬f⊆𝒬\\mathcal\{Q\}\_\{f\}\\subseteq\\mathcal\{Q\}\. We can represent the feature valuef​\(d\)∈𝒱ff\(d\)\\in\\mathcal\{V\}\_\{f\}as the set of answers to𝒬f\\mathcal\{Q\}\_\{f\}for documentdd\(see Step 2 in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)\)\.

##### Genie

TheGenienovelty ofdtd\_\{t\}is defined as avectorthat consists of the individual novelty scores along each featuref∈ℱf\\in\\mathcal\{F\}:

𝐆​\(dt\)=\[Gfeat​\(dt,f\)\]f∈ℱ\\mathcal\{\\mathbf\{G\}\}\(d\_\{t\}\)=\[\\text\{G\}\_\{\\text\{feat\}\}\(d\_\{t\},f\)\]\_\{f\\in\\mathcal\{F\}\}
The novelty of the target document alongffis defined as the average novelty across all questionsq∈𝒬fq\\in\\mathcal\{Q\}\_\{f\}:

Gfeat​\(dt,f\)=1\|𝒬f\|​∑qi∈𝒬fGquest​\(dt,qi\)\\text\{G\}\_\{\\text\{feat\}\}\(d\_\{t\},f\)=\\frac\{1\}\{\|\\mathcal\{Q\}\_\{f\}\|\}\\sum\_\{q\_\{i\}\\in\\mathcal\{Q\}\_\{f\}\}\\text\{G\}\_\{\\text\{quest\}\}\(d\_\{t\},q\_\{i\}\)
WhereGquest​\(dt,qi\)\\text\{G\}\_\{\\text\{quest\}\}\(d\_\{t\},q\_\{i\}\)\(i\.e\., the novelty ofdtd\_\{t\}alongqq\) is defined as the average pairwise dissimilarity \(1−1\-similarity\) between the target answeraaand the corresponding answers in each population document \(see Step 3 in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)\)\. Letadta\_\{d\_\{t\}\}andadpa\_\{d\_\{p\}\}be answers to questionqqfound in target documentdtd\_\{t\}and population documentdp∈𝒟p​o​pd\_\{p\}\\in\\mathcal\{D\}\_\{pop\}, respectively\.

Gquest​\(dt,q\)=1\|𝒟p​o​p\|​∑dp∈𝒟p​o​pdissim​\(adt,adp\)\\text\{G\}\_\{\\text\{quest\}\}\(d\_\{t\},q\)=\\frac\{1\}\{\|\\mathcal\{D\}\_\{pop\}\|\}\\sum\_\{d\_\{p\}\\in\\mathcal\{D\}\_\{pop\}\}\\mathrm\{dissim\}\(a\_\{d\_\{t\}\},a\_\{d\_\{p\}\}\)
Intuitively,dtd\_\{t\}will get a higherGeniescore on featureffif that feature is distinctive compared to howffis realized among the population documents\. In the example shown in Figure[2](https://arxiv.org/html/2606.12790#S1.F2),dtd\_\{t\}receives a higher score onsettingthanplotdue to its higher dissimilarity with the population ofsettings\. We keepGenieas a vector rather than an aggregate score to preserve fine\-grained explainability\. Additionally, the novelty of a response can be independently attributed to multiple features\.

## 4Geniefor Creative Writing

In this paper, we instantiateGenieon the creative writing task, in particular long\-form story generation\. We choose this task for several reasons: \(1\) it is known to be multi\-faceted with fine\-grained features such asplot,setting, etc\. that are researched in prior workHamilton et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib16)\); Russell et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib40)\)\. These existing features can validate the features that we automatically derived\. \(2\) Even though LLMs can generate coherent text, long\-form stories generated by LLMs are known to be uncreative or homogeneous\. Thus, it is a particularly valuable test bed for fine\-grained, explainable insights intowherethe \(lack of\) novelty appears, especially as existing metrics do not currently do so\.

### 4\.1Tasks

We curate a total of ~10,000 documents using 40 models, across 50 open\-ended writing prompts \(PP\) gathered from two sources: \(1\) 25 synthetic prompts generated bygpt\-4\.1\-minifollowing the formatWrite a story about X and YwhereXandYare spatially or temporally distant concepts \(Prompt[F](https://arxiv.org/html/2606.12790#A6)\) and \(2\) 25 prompts sampled from the LitBenchFein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\)subset of ther/WritingPromptsdatasetFan et al\. \([2018](https://arxiv.org/html/2606.12790#bib.bib10)\)\. Examples of the prompts in our dataset are in Appendix[A\.4](https://arxiv.org/html/2606.12790#A1.SS4)\. A power analysis reveals the number of prompts is a sufficiently large sample size for our experiments to achieve statistical significance \(Appendix[A\.1](https://arxiv.org/html/2606.12790#A1.SS1)\)\.

### 4\.2Features

We gathered features describing the creative writing task from promptinggpt\-4\.11010times \(Prompt[F\.1](https://arxiv.org/html/2606.12790#A6.SS1)\) eliciting a definition for each, and clustered the responses using embeddings computed withtext\-embedding\-3\-small\.111HBDSCAN, Agglomerative and K\-means clustering resulted in similar clusters; we setkkto the average number of features in each sample\.The features discovered include:setting, plot, characterization, mood/tone, point of view, language, theme, which closely correspond to the taxonomy described inHamilton et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib16)\)\. We therefore adopt their terminology and map our discovered features accordingly\. In this work, we specifically focus onagent,perspective,plot,setting,social atmosphere,222We renamedSocial Networkas used in NarraBench toSocial Atmospheredue to the former’s loaded meaning\.andstyleto form the setℱ\\mathcal\{F\}\. Details on how the features and definitions we discovered were mapped are included in Appendix[A\.2](https://arxiv.org/html/2606.12790#A1.SS2)\.

### 4\.3Question Generation

For each promptp∈Pp\\in P, we \(a\) generate the set of questions𝒬\\mathcal\{Q\}\(§[3\.2](https://arxiv.org/html/2606.12790#S3.SS2.SSS0.Px1)\), \(b\) perform filtering, and \(c\) map each question to the most representative feature inℱ\\mathcal\{F\}\. Each stage of the pipeline usesgpt\-4\.1\-mini\. The prompts are in Appendix[F\.2](https://arxiv.org/html/2606.12790#A6.SS2)\.

##### Filtering

Questions undergo a one\-step filtering stage to exclude questions that are polar, hypothetical, conditional, subjective or those that address multiple features\. Since the goal of using questions is to extract features and view them as independently as possible \(i\.e\. the novelty ofsettingis not dependent or influenced by the novelty ofstyle\)\.

##### Feature Mapping

We then map𝒬\\mathcal\{Q\}toℱ\\mathcal\{F\}as described in §[3\.2](https://arxiv.org/html/2606.12790#S3.SS2.SSS0.Px2)\. This results in𝒬f\\mathcal\{Q\}\_\{f\}for all six creative writing features\. Table[1](https://arxiv.org/html/2606.12790#S4.T1)shows statistics on the distribution of questions generated\.agentandplotare the largest features in question coverage, owing to the richness of the content that can be considered as either the storyline or character development\.plotrequires more questions to cover the conflict, turning points, obstacles, plotline, etc\. On the other hand,perspectivequestions are largely limited to extractingpoint of viewrelated information\.

Table 1:Distribution of questions generated per feature and per prompt\.plotandagentfeatures are represented by more questions, per prompt, due to their higher complexity relative toperspective\.

### 4\.4Constructing the Population and Target Set

##### Models

Using prompts from Section[4\.1](https://arxiv.org/html/2606.12790#S4.SS1), we build apopulationconsisting of 21 models ranging across various families, sizes and stages of training \(Table[16](https://arxiv.org/html/2606.12790#A8.T16)\)\. We consider 18targetmodels \(Table[17](https://arxiv.org/html/2606.12790#A8.T17)\)\. The target models roughly correspond to newer iterations of the population models so thatGeniecan reflect novelty differences as models evolve\.

##### Responses

For each prompt, we sample 5 responses from every model, across the population and target models \(see Step 2 in Figure[2](https://arxiv.org/html/2606.12790#S1.F2)\)\. All responses are post\-processed to truncate incoherent or irrelevant responses, and reasoning\. We then use ArmoRMWang et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib48)\)to compute coherence and further discard documents that do not meet a minimum threshold of 2\.0\.333Completely incoherent documents can have artificially high novelty and thus must be filtered out\. This threshold is chosen after manual inspection on a sample of documents\.

Thetargetset for which we calculate novelty consists of 4500 documents\. We use a pilot dataset and conduct a power analysis to determine the statistically significant population size required to reach 98% confidence with a0\.10\.1margin of error\. The resultingpopulationcontains 3404 documents\. Details about the sampling process are in Appendix[A\.1](https://arxiv.org/html/2606.12790#A1.SS1)\.

### 4\.5Dissimilarity Function

To calculatedissim​\(adt,adp\)\\mathrm\{dissim\}\(a\_\{d\_\{t\}\},a\_\{d\_\{p\}\}\), we judge similarity usingQwen2\.5\-32B\-Instructon a 4\-point Likert scale using the rubric shown in[G](https://arxiv.org/html/2606.12790#A7)and compute1−similarity1\-\\mathrm\{similarity\}\. Two answers are considered highly similar if they are interchangeable, despite slightly different wording\. On the other hand, dissimilar answers have very low overlap in the details described, and sometimes can contradict each other\. Pairs that are difficult to judge often have overlapping content or scope\. Some questions are unanswerable by certain documents \(“unspecified”, “not applicable”, etc\.\)\. We exclude these answers to preventGeniescores from rewarding unanswerability\.

## 5EvaluatingGenie

We intrinsically evaluate components ofGenie, such as question generation, feature mapping and the dissimilarity function, with the help of human annotators \(recruitment details are provided in[C\.2](https://arxiv.org/html/2606.12790#A3.SS2)\)\. We then conduct a study to measureGenie’ssensitivityto specific features and itsrobustnessto surface\-level variation\.

### 5\.1Intrinsic Validation

##### Question Generation

We run a small scale human study with two annotators to evaluate the reliability of using an LLM for generating questions given a prompt \(§[3\.2](https://arxiv.org/html/2606.12790#S3.SS2.SSS0.Px1)\)\. The annotators are give5050questions and are asked to evaluate the validity of each question using the criteria given in Appendix[G](https://arxiv.org/html/2606.12790#A7)\. The inter\-annotator agreement \(Krippendorff\-α\\alpha\) is0\.59\\mathbf\{0\.59\}\(moderately high,Artstein and Poesio \([2008](https://arxiv.org/html/2606.12790#bib.bib1)\)\)\. Some common source of disagreements are listed in Appendix[C\.1](https://arxiv.org/html/2606.12790#A3.SS1)\. The question generatorgpt\-4\.1\-miniachieves an F1 score of0\.71\\mathbf\{0\.71\}and0\.74\\mathbf\{0\.74\}as evaluated by two annotators\.

##### Feature Mapping

Since reliable question\-to\-feature mapping is essential for feature coverage and low noise, we validate the mapping step in §[3\.2](https://arxiv.org/html/2606.12790#S3.SS2.SSS0.Px2)\. Two annotators classified a set of 50 questions into one of the six features, or none, as given in §[4](https://arxiv.org/html/2606.12790#S4)using the criteria in Appendix[G](https://arxiv.org/html/2606.12790#A7)\. The Krippendorff\-α\\alphawas high at0\.85\\mathbf\{0\.85\}, and as shown in Table[2](https://arxiv.org/html/2606.12790#S5.T2),gpt\-4\.1\-miniachieves high F1 scores across all features against both annotators\.

##### Similarity

To validate the LLM dissimilarity function, we asked human annotators to judge the similarity of3,0003,000randomly sampled pairs of answers according to guidelines in Appendix[G](https://arxiv.org/html/2606.12790#A7)\. The annotation set is selected to maximize coverage across features and target\-population model pairs\. Each pair of answers is annotated by two annotators\. Table[11](https://arxiv.org/html/2606.12790#A3.T11)reports per\-feature agreement between annotators, as well as between annotators andQwen2\.5\-32B\-Instruct\(the LLM used for distance judgments\)\. Measured with Krippendorff\-α\\alpha, inter\-annotator agreement is0\.59\\mathbf\{0\.59\}, and higher0\.66\\mathbf\{0\.66\}for annotators vs\. LLMs\.

Table 2:Model \(gpt\-4\.1\-mini\) accuracy for question\-to\-feature mapping\.

### 5\.2GenieSensitivity to Feature Edits

We conduct a causal intervention to investigate thesensitivityofGenieto differences between responses along specific features\.Sensitivityis defined as the ability to detect a statistically significant change inGenie’s observed novelty for featureffwhen an intervention is performed alongff\.Robustnessrefers to statistical equivalence in the observed novelty for featureffwhen a lexical edit, such as paraphrasing, is performed\. An ideal metric for measuring content\-level novelty detects all interventions \(high sensitivity\), while not responding to paraphrases \(high robustness\)\.

To test for sensitivity, we surgically edit documents along individual features using population\-agnostic synthetic alternatives, while maximally preserving other features\. To test for robustness, we paraphrase documents in a semantics\-preserving manner, altering surface form while preserving all content features\.

#### 5\.2\.1Experimental Setup

##### Feature Intervention

We first define an edit functionE:𝒟×ℱ×𝒱→𝒟E:\\mathcal\{D\}\\times\\mathcal\{F\}\\times\\mathcal\{V\}\\rightarrow\\mathcal\{D\}that produces a minimally edited documentd′=E​\(d,fi,v~fi\)d^\{\\prime\}=E\(d,f\_\{i\},\\tilde\{v\}\_\{f\_\{i\}\}\), wherefi∈ℱf\_\{i\}\\in\\mathcal\{F\}is the feature being intervened upon andv~fi\\tilde\{v\}\_\{f\_\{i\}\}is the new proposed value offif\_\{i\}\. The original documentddand the edited documentd′d^\{\\prime\}differ along the value offif\_\{i\}while maximally preserving all else\. Concretely, we synthetically generate an alternate valuev~fi\\tilde\{v\}\_\{f\_\{i\}\}for the target feature using an LLM, conditioned onddand the promptppbut independent of any population \(Prompt[F\.3](https://arxiv.org/html/2606.12790#A6.SS3)\);d′d^\{\\prime\}is then the rewrite ofddthat reflectsv~fi\\tilde\{v\}\_\{f\_\{i\}\}, with allotherfeature values held fixed \(i\.e\.,∀f≠fi:f​\(d\)=f​\(d′\)\\forall f\\neq f\_\{i\}:f\(d\)=f\(d^\{\\prime\}\)\)\. This helps ensure that there is a meaningful difference alongfif\_\{i\}betweenddandd′d^\{\\prime\}without a predefined notion of relative novelty\. We sample alternate features usinggpt\-5\.4\-mini\(Prompt[F\.3](https://arxiv.org/html/2606.12790#A6.SS3)\) and implement the edits usinggpt\-4\.1\-mini\(Prompt[F\.3](https://arxiv.org/html/2606.12790#A6.SS3)\)\.444Stronger models are not suitable for generating minimal edits required for this studyLaban et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib23)\)\.

##### Dataset

We randomly sample100100instruct\-model generated documents from the target dataset\. For each document and along each feature, we generate 3 alternatives and perform edits to produce 3 edited documents\. This results in18001800total pairs of original and edited documents, of which we filtered out206206due to malformed responses \(details provided in Appendix[B\.3](https://arxiv.org/html/2606.12790#A2.SS3)\)\.

##### Paraphrasing Documents

We form33paraphrases per document, resulting in300300additional pairs\. Semantically equivalent paraphrasing should not alter feature values\. Thus, a metric that finds a significant difference between paraphrased texts may be capturing other properties \(lexical, aesthetic qualities, etc\.\)\.

##### Intervention Validation

We validate the edit function along three main criteria: \(1\) The proposed alternative featurev~fi\\tilde\{v\}\_\{f\_\{i\}\}isvalidifv~fi∈𝒱fi\\tilde\{v\}\_\{f\_\{i\}\}\\in\\mathcal\{V\}\_\{f\_\{i\}\}\(i\.e\. the proposed alternative must reflect the intended intervention\); \(2\) An edit iseffectiveiffi​\(d′\)=v~fif\_\{i\}\(d^\{\\prime\}\)=\\tilde\{v\}\_\{f\_\{i\}\}\(i\.e\. the edits fully implement the proposed alternative such that it is extracted as the new feature value; \(3\) Edits areminimally invasiveif∀f≠fi:f​\(d′\)=f​\(d\)\\forall f\\neq f\_\{i\}:f\(d^\{\\prime\}\)=f\(d\)\. For instance, editing a story to be written from a differentperspectivecan lead to portions of the narrative being omitted for cohesion, thus dramatically impacting which characters or plot points are included\.

The authors annotated a set of 270 pairs of documents along this criteria on a 4\-point Likert scale using the rubric described in Appendix[G](https://arxiv.org/html/2606.12790#A7)\)\. Table[12](https://arxiv.org/html/2606.12790#A3.T12)shows the inter\-annotator agreement, measured using Randolph’s\-κ\\kappa\(Randolph,[2005](https://arxiv.org/html/2606.12790#bib.bib37)\)\.555This task did not have a fixed rating distribution prior to annotations\.Agreement is high for judging validity, and moderately high for determining effectiveness\. Judging invasiveness is subjective\. Thus, we measure the proportion of alternatives that are valid, edits that are effective and minimally invasive by computing the majority vote across annotators \(Table[13](https://arxiv.org/html/2606.12790#A3.T13)\)\. We discover that the edit function generates reliably valid and effective interventions\. We note that these edits are invasive forplot,social atmosphereandperspectiveinterventions, suggesting inherent inter\-feature dependence\.

![Refer to caption](https://arxiv.org/html/2606.12790v1/figures/genie_all_scores.png)Figure 3:Mean deltas registered byGeniefor each intervention along each feature\. \(\*\): Statistical significance\.Genieis most sensitive toperspectiveandsettinginterventions, and registers the largest delta for featurefif\_\{i\}whenfif\_\{i\}is intervened upon, except foragentandstyle\.

#### 5\.2\.2Sensitivity

We test whether intervening onfif\_\{i\}produces a systematic change inGenie’s score alongfif\_\{i\}, for allfi∈ℱf\_\{i\}\\in\\mathcal\{F\}\. We perform a two\-sided paired bootstrap test to find significance between pre\- and post\-intervention score distributions\. For paraphrases, equivalence is tested using two one\-sided tests\. Refer to Appendix[B](https://arxiv.org/html/2606.12790#A2)\.

##### Genieis sensitive to interventions

Figure[3](https://arxiv.org/html/2606.12790#S5.F3)shows the mean deltas registered byGeniepre\- and post\- intervention, per feature\.Genieaccurately registers a statistically significant difference for all feature interventions, which shows that it can detect differences in novelty\. Moreover, the heatmap indicates that when we intervene onperspective,plot,settingandsocial atmosphere,Genienot only detects the change, but also registers the largest mean delta along the intervened feature relative to all other feature scores\.

##### Genieis robust against paraphrases

The final row in Figure[3](https://arxiv.org/html/2606.12790#S5.F3)shows the mean deltas for paraphrased pairs\. The mean deltas are all smaller than the deltas registered for interventions \(diagonal values\)\. The two one\-sided paired bootstrap tests indicate that theGeniescores across all features for paraphrased documents are statistically equivalent\. This indicatesGenie’s robustness towards surface\-level properties of texts\.

## 6Evaluating Holistic Metrics

Existing metrics that measure novelty, diversity and creativity are generally holistic, i\.e\., they generate one number for a given input, making it unclear what aspects of novelty they are measuring\. ButGenieallows us to answer this question\. Specifically, we use a suite of baseline metrics to analyze their sensitivity in addition toGenie\. We use the same setup as in §[5\.2](https://arxiv.org/html/2606.12790#S5.SS2)\.

### 6\.1Population\-based Metrics

We measure the sensitivity of five holistic metrics that address novelty from different angles\.

Cosine Distance:We compute the average pairwise cosine distance of document embeddings usingtext\-embedding\-vs\-large\.

Compression Ratio \(CR\):We calculate the compression ratio of the population with and without the target response usinggzipto capture lexical diversityShaib et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib42)\)\.

Creativity Index \(Cr Idx\):We measure the semanticnn\-gram novelty forn≥Ln\\geq LfollowingLu et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib26)\)relative to a reference corpus\.

NoveltyBench Partition Model \(NB\):We utilize the partition model fromZhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\), which clusters candidate responses based on their functional utility where utility is defined as how beneficial a response is in terms of information value\.

Litbench GenRM \(LB\):We use theQwen\-2\.5\-7b\-Instructgenerative reward model fine\-tuned onFein et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib11)\)to calculate the average pairwise preference win\-rates\.

All metrics are instantiated with the population used byGenie\. For instance, the corpora used by CreativityIndex and the candidates considered by the partition model in NoveltyBench are both instantiated using our population\.

Geniev\. Holistic Metrics Effect Size \(Cohen’s d\)\*p<0\.008p<0\.008,p†<0\.008\{\}^\{\\dagger\}p<0\.008InterventionGCos\.CRCr IdxNBLBagent0\.62\*0\.54\*0\.051\.06\*0\.52\*\-0\.45\*perspective1\.01\*0\.67\*\-0\.19\*1\.17\*0\.14\-0\.87\*plot0\.71\*0\.22\*0\.010\.93\*0\.30\*\-0\.33\*setting0\.93\*0\.81\*1\.08\*0\.97\*0\.44\*\-0\.48\*soc\. atm\.0\.47\*0\.24\*0\.23\*1\.00\*0\.16\-0\.42\*style0\.38\*0\.68\*0\.071\.43\*0\.07\-1\.53\*Paraphrase0\.03†0\.45†1\.25†1\.290\.69\-0\.94Table 3:Effect sizes \(Cohen’sdd\) for feature interventions and paraphrases\.Genieand Cosine Distance are sensitive to all interventionsandparaphrase robust\. However, Cosine Distance weakly detectsplotandsocial atmosphere\. LitBench uniquely finds negative effect sizes, scoring intervened documents poorly\. Refer to Table[14](https://arxiv.org/html/2606.12790#A3.T14)forGenie’s paraphrase effect sizes \(note that the average is reported here\)\.
### 6\.2What Do Holistic Metrics Capture?

We calculate both the effect sizes \(Cohen’sdd\) and statistical significance for scores found using each metric before and after intervention\. This is shown in Table[3](https://arxiv.org/html/2606.12790#S6.T3)\. We note that the standard deviations vary across metrics, and report them in Table[9](https://arxiv.org/html/2606.12790#A2.T9)\.

##### Sensitivity

Cosine Distance, Creativity Index and LitBench are sensitive to all feature interventions\. Compression Ratio’s exclusive sensitivity tosettinginterventions, which use highly descriptive terminology and out\-of\-distributionnn\-grams \(e\.g\.rotating market pavilion on a lunar colony;glass factory built inside a dormant volcanic caldera\), indicates that it does not capture most types of content novelty\. NoveltyBench detects interventions moderately across most features, exceptperspective,social atmosphereandstyle\. While most metrics perceive edited documents as more novel, LitBench scores them lower with negative effect sizes; we discuss this in a later paragraph\.

##### Robustness

The final row in Table[3](https://arxiv.org/html/2606.12790#S6.T3)shows Cohen’sddmeasured when using metrics on paraphrases\. Values marked with†\\daggerdenote distributions that are statistically equivalent \(we use two one\-sided tests, see Appendix[B\.2](https://arxiv.org/html/2606.12790#A2.SS2)\)\. Creativity Indiex, NoveltyBench and LitBench do not demonstrate robustness\. Creativity Index appears to distinguish many types of interventions, but its lack of paraphrase robustness may indicate that it is overly sensitive to lexical novelty\.

Figure[4](https://arxiv.org/html/2606.12790#S6.F4)shows the min\-max normalized mean absolute deltas for interventions versus paraphrases for each metric and feature intervention\. Metrics in the shaded region are not robust as they detect higher average paraphrase deltas than for feature interventions\.Genieis the most consistently robust metric across all features\. Compression Ratio and NoveltyBench consistently register a larger difference between paraphrased documents than intervened documents\.

![Refer to caption](https://arxiv.org/html/2606.12790v1/figures/sensitivity_v_paraphrase_horizontal.png)Figure 4:Normalized Mean Sensitivity and Paraphrase Absolute Deltas for each metric, per intervention\. The shaded region represents higher average paraphrase sensitivity over interventions\.Genieis consistently robust across all interventions\. Holistic metrics, except Cosine Distance, are usually as equally or more sensitive to paraphrases than interventions\.
##### Aesthetic Quality

LitBench consistently registers negative effect sizes across all interventions and paraphrases\. Upon close inspection, we see that LitBench is capturing the degradation of a property consistent across edits\. A qualitative analysis revealed that interventions, though accurate \(Table[13](https://arxiv.org/html/2606.12790#A3.T13)\), are artificial and forced at times \(e\.g\.perspectiveinterventions include rewriting the story fromfirst\-person plural from the collective voice of the canned goods\)\. Thus, we note that the palatability of edited documents may be degraded, and correlated with LitBench\. Although novelty and quality have a trade\-off relation in model\-generated responsesWest and Potts \([2025](https://arxiv.org/html/2606.12790#bib.bib50)\), they are independent axes of creativityBoden \([1991](https://arxiv.org/html/2606.12790#bib.bib3)\)\. Thus, metrics that capture aesthetic value are not suitable novelty metrics\.

##### Functional Utility

Since the NoveltyBench partition model was optimized for short\-context tasks rather than long\-form creative writing, it is unclear which properties of a response determine its functional utility\. However, the features it does not capture \(perspective,social atmosphereandstyle\) are properties that are peripheral to the document \(i\.e\. not explicitly stated in the document\)\. Thus, we hypothesize that functional utility correlates with, and may be capturing, visible properties, lexical or semantic\.

##### Spurious Correlations

Table[4](https://arxiv.org/html/2606.12790#S6.T4)shows the raw mean scores calculated by Cosine Distance andGenieper intervention\. For certain interventions, such asplot,Genieappears to register a more dramatic change in novelty\. Cosine Distance registers the same delta forperspectiveandstyleinterventions \(0\.050\.05\), whileGenieregisters a very large delta forperspective, but notstyle\. This suggests that Cosine Distance andGenieare capturing different properties \(refer to Table[15](https://arxiv.org/html/2606.12790#A5.T15)for a correlation study\)\.

Table 4:The raw mean scores for Cosine Distance andGeniepre\- and post\- intervention\.

## 7Effectiveness of Mitigation Methods

The reduced diversity and creativity in model\-generated responsesKirk et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib22)\); Zhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\); Jiang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib19)\)has motivated work that aims to mitigate this behavior and improve various dimensions of creativity\. Some of these methods are evaluated using holistic metrics, such as pairwise cosine similarity and lexical metricsZhang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib54)\); Peeperkorn et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib35)\), as well as human and/or LLM judgesHuot et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib17)\); Chakrabarty et al\. \([2025b](https://arxiv.org/html/2606.12790#bib.bib8)\); Pham et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib36)\)\. We useGenieto investigate the effectiveness of a subset of such methods\. Specifically, we measure the distributional differences observed pre\- and post\- mitigation\. For the following experiments, we aggregate theGeniescores over all generations per prompt to form paired data\.

### 7\.1Target Methods

We consider two broad categories of mitigation methods: alternate prompting and decoding based methods\.

We evaluate three prompting strategies:\(1\) Verbalized Sampling\(VS\) refers to generatingnsamples using the target modelMMwith their associated probabilitiesZhang et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib54)\)\.\(2\)We use Agents’ Room style promptingHuot et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib17)\)to form detailedOutlinesfor each prompt and generate a document given the outline\. This outlining requires the model to explicitly plan the Setting, Characters, Plot, etc\., which closely follow the features of creative writing we use\.\(3\)Negative Sampling\(NS\) is defined as generating a documentddusing promptppand addingddto the context and prompting the model to generate a responsedifferentfromddZhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\)\.

Finally, we exploreConformative Decoding\(CD\), a decoding strategy that blends base and instruct models at the token\-levelPeeperkorn et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib35)\)\. By using the weighted sum of token probabilities, the instruct model helps improve the quality while the base model introduces diverse content\.666We weight base models higher than instruct models \(0\.6 and 0\.4, respectively\) to inspire more diverse outputs\.

Using the methods above, we generate responses to the prompts mentioned in §[4\.1](https://arxiv.org/html/2606.12790#S4.SS1)for the following models:gemma\-3\-12b\-it,gemma\-3\-27b\-it,Llama\-3\.1\-8B\-Instruct,OLMo\-3\-7B\-InstructandOLMo\-3\-32B\-Instruct\. Using these methods, we sample55responses for each prompt, per model\. We then calculateGenieto study the fine\-grained improvements mitigation helps bring\.

### 7\.2Results

Geniescores for each feature are aggregated across generations of documents per prompt, per model\. For Negative Sampling, specifically, we pair the data at the instance level since target responses are paired prior to generating the data \(adding the response to the context\)\. We perform a paired bootstrap two\-sided test betweenGeniescores for responses generated pre\- and post\- mitigation\. The meanGeniedeltas are reported in in Table[5](https://arxiv.org/html/2606.12790#S7.T5), with significance marked\.

Outlining helps improveperspectiveandstyle, while Conformative Decoding improvessetting\. Negative Sampling has a significant improvement relative to the original counterpart\. This is consistent with the findings ofZhang et al\. \([2025](https://arxiv.org/html/2606.12790#bib.bib55)\)regarding the effectiveness of this method\. We note that despite significance, all mean deltas are small \(e\.g\. relative toGeniedetecting a delta of0\.350\.35onperspectiveinterventions in Figure[3](https://arxiv.org/html/2606.12790#S5.F3)\)\.

Although Verbalized Samplingappearsto be the most effective mitigation method, Table[5](https://arxiv.org/html/2606.12790#S7.T5)shows that responses generated by this method are disproportionately shorter relative to the target and population set\. These lengths \(∼\\sim100 words\) are similar to flash fiction with22\-33sentences, rather than a fully formed story \(e\.g\. Appendix[E](https://arxiv.org/html/2606.12790#A5)\)\. Thus, comparing the novelty of these documents with respect to a population that is from a different distribution is not representative\. This highlights the importance of using an appropriate population when computing novelty\.

This analysis illustrates what fine\-grained evaluation enables, rather than a single aggregate score,Genieattributes each method’s effect to specific features\.

\.

Table 5:MeanGeniedeltas observed when using mitigation methods\. Significance is marked with \(\*\)\. Conformative Decoding helps improvesettingand Outlinging improvesperspectiveandstyle\. The average number of words in Verbalized Sampling scores are not representative as responses are disproportionately small relative to the population \(488\) and target \(513\) set\. Average Coherence computed using ArmoRM \(out of 5\) is moderate across all methods\.

## 8Conclusion

We presentGenie, a fine\-grained evaluation metric to measure thenoveltyof LLM\-generated responses with respect to an explicitly defined population\.Geniediscovers and grounds task\-specificfeaturesin responses using answers to prompt\-specific questions\. We instantiateGeniein the creative writing domain as a testbed and show thatGenieis sensitive to features when independently intervened upon and robust towards lexical edits \(i\.e\. paraphrases\)\. UsingGenie, we study the sensitivity and robustness of existing holistic creativity, diversity and novelty and find that they either capture extraneous properties or are lexically sensitive\. Finally, we evaluate the effectiveness of mitigation methods using a fine\-grained lens, revealing the strengths of alternate prompting and decoding strategies\.

## Limitations

##### Computational Cost

TheGeniepipeline is resource\-intensive\. We use proprietary models for feature discovery, question generation, question\-to\-feature mapping, alternate feature generation, document editing, and paraphrasing, andQwen2\.5\-32B\-Instructfor question answering and similarity judgments\. UsingGeniefor other domains requires generating new populations becauseGenieis heavily task\-oriented\. That said, measuring novelty is really difficult, and we view the compute cost worth the cost of going beyond the coarse, surface\-level proxies to explainable, feature\-grounded evaluation \(see Appendix[A\.5](https://arxiv.org/html/2606.12790#A1.SS5)for more details\)\.

##### Domain Scope

We instantiateGeniein the creative writing domain\. The pipeline is, by design, generalizable to any text\-based task but we have not run extensive experiments measuring how well existing metrics capture novelty outside creative writing\. Thus, this pipeline has not been validated for use in domains such as scientific writing, code or argumentative essays\.

## Acknowledgments

We thank Kathryn Kazanas for helping with preliminary annotations and the TAUR Lab for providing feedback on the project\. This work was partially supported by NSF grants IIS\-2107524, IIS\-2145479, IIS\-2145280, and IIS\-2433071, as well as the Sloan Foundation\. Compute for this project was partially supplied by the Machine Learning Lab at UT Austin and the NSF AI Institute for Foundations of Machine Learning \(IFML\)\.

## References

- Artstein and Poesio \(2008\)Ron Artstein and Massimo Poesio\. 2008\.[Inter\-Coder Agreement for Computational Linguistics](https://doi.org/10.1162/coli.07-034-R2)\.*Computational Linguistics*, 34\(4\):555–596\.
- Beaver et al\. \(2017\)David I\. Beaver, Craige Roberts, Mandy Simons, and Judith Tonhauser\. 2017\.Questions Under Discussion: Where Information Structure Meets Projective Content\.*Annual Review of Linguistics*, 3\(1\):265–284\.
- Boden \(1991\)Margaret A\. Boden\. 1991\.*The Creative Mind: Myths and Mechanisms*\.Basic Books, Inc\., USA\.
- Campello et al\. \(2013\)Ricardo J\. G\. B\. Campello, Davoud Moulavi, and Joerg Sander\. 2013\.Density\-Based Clustering Based on Hierarchical Density Estimates\.In*Advances in Knowledge Discovery and Data Mining*, pages 160–172, Berlin, Heidelberg\. Springer Berlin Heidelberg\.
- Cao et al\. \(2026\)Qian Cao, Xiting Wang, Yuzhuo Yuan, Yahui Liu, Fang Luo, and Ruihua Song\. 2026\.[Evaluating Text Creativity across Diverse Domains: a Dataset and Large Language Model Evaluator](https://openreview.net/forum?id=WHrmlyQEM7)\.In*The Fourteenth International Conference on Learning Representations*\.
- Chakrabarty et al\. \(2024\)Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien\-Sheng Wu\. 2024\.[Art or Artifice? Large Language Models and the False Promise of Creativity](https://doi.org/10.1145/3613904.3642731)\.In*Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*, CHI ’24, New York, NY, USA\. Association for Computing Machinery\.
- Chakrabarty et al\. \(2025a\)Tuhin Chakrabarty, Philippe Laban, and Chien\-Sheng Wu\. 2025a\.[AI\-Slop to AI\-Polish? Aligning Language Models through Edit\-Based Writing Rewards and Test\-time computation](https://openreview.net/forum?id=jeDYcjuZIV)\.In*Second Conference on Language Modeling*\.
- Chakrabarty et al\. \(2025b\)Tuhin Chakrabarty, Philippe Laban, and Chien\-Sheng Wu\. 2025b\.[Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human\-AI Alignment in the Writing Process through Edits](https://doi.org/10.1145/3706598.3713559)\.In*Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*, CHI ’25, New York, NY, USA\. Association for Computing Machinery\.
- Ettinger et al\. \(2026\)Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, and 48 others\. 2026\.[Olmo 3](https://arxiv.org/abs/2512.13961)\.*Preprint*, arXiv:2512\.13961\.
- Fan et al\. \(2018\)Angela Fan, Mike Lewis, and Yann Dauphin\. 2018\.Hierarchical Neural Story Generation\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 889–898\.
- Fein et al\. \(2026\)Daniel Fein, Sebastian Russo, Violet Xiang, Kabir Jolly, Rafael Rafailov, and Nick Haber\. 2026\.[LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://doi.org/10.18653/v1/2026.eacl-long.362)\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 7740–7755, Rabat, Morocco\. Association for Computational Linguistics\.
- Franceschelli and Musolesi \(2024\)Giorgio Franceschelli and Mirco Musolesi\. 2024\.[On the Creativity of Large Language Models](https://doi.org/10.1007/s00146-024-02127-3)\.*AI Soc\.*, 40\(5\):3785–3795\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others\. 2024\.[The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Hamblin \(1957\)Charles Leonard Hamblin\. 1957\.*Language and the Theory of Information*\.Ph\.D\. thesis, London School of Economics and Political Science \(University of London\)\.
- Hamilton \(2024\)Sil Hamilton\. 2024\.[Detecting Mode Collapse in Language Models via Narration](https://doi.org/10.18653/v1/2024.scalellm-1.5)\.In*Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models \(SCALE\-LLM 2024\)*, pages 65–72, St\. Julian’s, Malta\. Association for Computational Linguistics\.
- Hamilton et al\. \(2026\)Sil Hamilton, Matthew Wilkens, and Andrew Piper\. 2026\.[NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://doi.org/10.18653/v1/2026.eacl-long.176)\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3786–3801, Rabat, Morocco\. Association for Computational Linguistics\.
- Huot et al\. \(2025\)Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, and Mirella Lapata\. 2025\.[Agents’ Room: Narrative Generation through Multi\-step Collaboration](https://openreview.net/forum?id=HfWcFs7XLR)\.In*The Thirteenth International Conference on Learning Representations*\.
- Ismayilzada et al\. \(2025\)Mete Ismayilzada, Antonio Laverghetta Jr\., Simone A\. Luchini, Reet Patel, Antoine Bosselut, Lonneke Van Der Plas, and Roger E\. Beaty\. 2025\.[Creative Preference Optimization](https://doi.org/10.18653/v1/2025.findings-emnlp.509)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 9580–9609, Suzhou, China\. Association for Computational Linguistics\.
- Jiang et al\. \(2026\)Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi\. 2026\.[Artificial Hivemind: The Open\-Ended Homogeneity of Language Models \(and Beyond\)](https://openreview.net/forum?id=saDOrrnNTz)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track*\.
- Kamath et al\. \(2025\)Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean\-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gael Liu, and 191 others\. 2025\.[Gemma 3 Technical Report](https://api.semanticscholar.org/CorpusID:277313563)\.*ArXiv*, abs/2503\.19786\.
- Karttunen \(1977\)Lauri Karttunen\. 1977\.Syntax and Semantics of Questions\.*Linguistics and philosophy*, 1:3–44\.
- Kirk et al\. \(2024\)Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu\. 2024\.[Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://openreview.net/forum?id=PXD3FAVHJT)\.In*The Twelfth International Conference on Learning Representations*\.
- Laban et al\. \(2026\)Philippe Laban, Tobias Schnabel, and Jennifer Neville\. 2026\.[LLMs Corrupt Your Documents When You Delegate](https://api.semanticscholar.org/CorpusID:287607479)\.*arXiv preprint*\.
- Lahiri \(2001\)Utpal Lahiri\. 2001\.*Questions and Answers in Embedded Contexts*\.Oxford University Press UK\.
- Le Bronnec et al\. \(2024\)Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Alexandre Allauzen\. 2024\.[Exploring Precision and Recall to assess the quality and diversity of LLMs](https://doi.org/10.18653/v1/2024.acl-long.616)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 11418–11441, Bangkok, Thailand\. Association for Computational Linguistics\.
- Lu et al\. \(2025\)Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi\. 2025\.[AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://openreview.net/forum?id=ilOEOIqolQ)\.In*The Thirteenth International Conference on Learning Representations*\.
- McQueen \(1967\)James B McQueen\. 1967\.Some methods of classification and analysis of multivariate observations\.In*Proc\. of 5th Berkeley Symposium on Math\. Stat\. and Prob\.*, pages 281–297\.
- Merrill et al\. \(2024\)William Merrill, Noah A\. Smith, and Yanai Elazar\. 2024\.[Evaluatingnn\-Gram Novelty of Language Models Using Rusty\-DAWG](https://doi.org/10.18653/v1/2024.emnlp-main.800)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 14459–14473, Miami, Florida, USA\. Association for Computational Linguistics\.
- Mesnard et al\. \(2024\)Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L\. Sifre, Morgane Rivière, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L’eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro\-Ros, Ambrose Slone, Am’elie H’eliou, Andrea Tacchetti, and 88 others\. 2024\.[Gemma: Open models based on gemini research and technology](https://api.semanticscholar.org/CorpusID:268379206)\.*ArXiv*, abs/2403\.08295\.
- Mojena \(1977\)Richard Mojena\. 1977\.Hierarchical grouping methods and stopping rules: an evaluation\.*The Computer Journal*, 20\(4\):359–363\.
- Nagarajan et al\. \(2025\)Vaishnavh Nagarajan, Chen Henry Wu, Charles Ding, and Aditi Raghunathan\. 2025\.[Roll the dice & look before you leap: Going beyond the creative limits of next\-token prediction](https://openreview.net/forum?id=Hi0SyHMmkd)\.In*Forty\-second International Conference on Machine Learning*\.
- Namuduri et al\. \(2025\)Ramya Namuduri, Yating Wu, Anshun Asher Zheng, Manya Wadhwa, Greg Durrett, and Junyi Jessy Li\. 2025\.[QUDsim: Quantifying Discourse Similarities in LLM\-Generated Text](https://openreview.net/forum?id=zFz1BJu211)\.In*Second Conference on Language Modeling*\.
- Padmakumar and He \(2024\)Vishakh Padmakumar and He He\. 2024\.[Does Writing with Language Models Reduce Content Diversity?](https://openreview.net/forum?id=Feiz5HtCD0)In*The Twelfth International Conference on Learning Representations*\.
- Padmakumar et al\. \(2026\)Vishakh Padmakumar, Chen Yueh\-Han, Jane Pan, Valerie Chen, and He He\. 2026\.[Measuring LLM Novelty As The Frontier Of Original And High\-Quality Output](https://openreview.net/forum?id=i7QNKZioN6)\.In*The Fourteenth International Conference on Learning Representations*\.
- Peeperkorn et al\. \(2025\)Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous\. 2025\.[Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction\-Tuned Large Language Models](https://doi.org/10.48550/arXiv.2507.20956)\.*CoRR*, abs/2507\.20956\.
- Pham et al\. \(2025\)Chau Minh Pham, Jenna Russell, Dzung Pham, and Mohit Iyyer\. 2025\.[Frankentext: Stitching random text fragments into long\-form narratives](https://api.semanticscholar.org/CorpusID:278886702)\.*ArXiv*, abs/2505\.18128\.
- Randolph \(2005\)Justus J\. Randolph\. 2005\.[Free\-Marginal Multirater Kappa \(multirater K \[free\]\): An Alternative to Fleiss’ Fixed\-Marginal Multirater Kappa](https://eric.ed.gov/?id=ED490661)\.Presented at the Joensuu Learning and Instruction Symposium\.ERIC Document ED490661\.
- Riviere et al\. \(2024\)Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, and 178 others\. 2024\.[Gemma 2: Improving Open Language Models at a Practical Size](https://arxiv.org/abs/2408.00118)\.*Preprint*, arXiv:2408\.00118\.
- Roberts \(2012\)Craige Roberts\. 2012\.Information Structure: Towards an integrated formal theory of pragmatics\.*Semantics and pragmatics*, 5:6–1\.
- Russell et al\. \(2026\)Jenna Russell, Rishanth Rajendhran, Chau Minh Pham, Mohit Iyyer, and John Wieting\. 2026\.[StoryScope: Investigating Idiosyncrasies in AI Fiction](https://api.semanticscholar.org/CorpusID:287122180)\.
- Schapiro et al\. \(2025\)Samuel Schapiro, Sumuk Shashidhar, Alexi Gladstone, Jonah Black, Royce Moon, Dilek Hakkani\-Tur, and Lav R Varshney\. 2025\.Combinatorial Creativity: A New Frontier in Generalization Abilities\.*arXiv preprint arXiv:2509\.21043*\.
- Shaib et al\. \(2024\)Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C\. Wallace\. 2024\.[Detection and Measurement of Syntactic Templates in Generated Text](https://api.semanticscholar.org/CorpusID:270869797)\.In*Conference on Empirical Methods in Natural Language Processing*\.
- Slocum et al\. \(2025\)Stewart Slocum, Asher Parker\-Sartori, and Dylan Hadfield\-Menell\. 2025\.[Diverse Preference Learning for Capabilities and Alignment](https://openreview.net/forum?id=pOq9vDIYev)\.In*The Thirteenth International Conference on Learning Representations*\.
- Touvron et al\. \(2023\)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others\. 2023\.[Llama 2: Open Foundation and Fine\-Tuned Chat Models](https://arxiv.org/abs/2307.09288)\.*Preprint*, arXiv:2307\.09288\.
- Velleman and Beaver \(2016\)Leah Velleman and David I\. Beaver\. 2016\.[Question\-based Models of Information Structure](https://doi.org/10.1093/oxfordhb/9780199642670.013.29)\.In Caroline Féry and Shinichiro Ishihara, editors,*The Oxford Handbook of Information Structure*, pages 86–107\. Oxford University Press, Oxford, UK\.
- Wadhwa et al\. \(2026\)Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, and Greg Durrett\. 2026\.CREATE: Testing LLMs for Associative Creativity \.*arXiv preprint arXiv:2603\.09970*\.
- Walsh et al\. \(2025\)Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, and 23 others\. 2025\.[2 OLMo 2 Furious](https://arxiv.org/abs/2501.00656)\.*Preprint*, arXiv:2501\.00656\.
- Wang et al\. \(2024\)Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang\. 2024\.Interpretable Preferences via Multi\-Objective Reward Modeling and Mixture\-of\-Experts\.In*EMNLP*\.
- Wenger and Kenett \(2025\)Emily Wenger and Yoed N\. Kenett\. 2025\.[We’re Different, We’re the Same: Creative Homogeneity Across LLMs](https://api.semanticscholar.org/CorpusID:276079902)\.*ArXiv*, abs/2501\.19361\.
- West and Potts \(2025\)Peter West and Christopher Potts\. 2025\.[Base Models Beat Aligned Models at Randomness and Creativity](https://openreview.net/forum?id=vqN8uom4A1)\.In*Second Conference on Language Modeling*\.
- Xu et al\. \(2025\)Weijia Xu, Nebojsa Jojic, Sudha Rao, Chris Brockett, and Bill Dolan\. 2025\.[Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://doi.org/10.1073/pnas.2504966122)\.*Proceedings of the National Academy of Sciences*, 122\(35\):e2504966122\.
- Yang et al\. \(2025a\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025a\.[Qwen3 technical report](https://api.semanticscholar.org/CorpusID:278602855)\.
- Yang et al\. \(2025b\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others\. 2025b\.[Qwen2\.5 Technical Report](https://arxiv.org/abs/2412.15115)\.*Preprint*, arXiv:2412\.15115\.
- Zhang et al\. \(2026\)Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael Tomz, Christopher D Manning, and Weiyan Shi\. 2026\.[Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://openreview.net/forum?id=9jQkmGunGo)\.
- Zhang et al\. \(2025\)Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito\. 2025\.[NoveltyBench: Evaluating Creativity and Diversity in Language Models](https://openreview.net/forum?id=XZm1ekzERf)\.In*Second Conference on Language Modeling*\.

## Appendix AGenie

### A\.1Building the Population

Genierequires a population or reference corpus\. For our experiments, to determine the population sample size required for statistical confidence, we performed a power analysis using a pilot set of 100 randomly sampled target documents across all 50 prompts\. We compute the similarity between each target document and the entire population\. Table[9](https://arxiv.org/html/2606.12790#A2.T9)shows the average variance and standard deviation of similarity scores across documents, for each feature\. The minimum population sample size required for each feature is reported in Table[9](https://arxiv.org/html/2606.12790#A2.T9)for 98% confidence and a margin of error of 0\.1\.

In order to sample a global population that is consistent across features, we use the maximum required size i\.e\. 50 documents\. We randomly sample these from the population \. Although the analysis determines the minimum number ofanswer\-pairsneeded for eachfeature, we sample at the document level in order to compare all metrics fairly instead of providing partial information to some\. Sampling at the document\-level will inherently produce a population that islargerthan required\.

We also conduct an retrospective analysis and check the power to see if the data is sufficiently large to capture the effect size that is observed\. The resulting dataset that we have has sufficient power to determine the statistical significance marked in Table[5](https://arxiv.org/html/2606.12790#S7.T5)\. However, this analysis is done per feature, and data is lost through theGeniepipeline \(filtering documents, unanswerable documents, etc\.\)\. We note that we achieve power≥0\.9\\geq 0\.9for cases where≥44\\geq 44prompt\-pairs survived\. Thus5050is sufficiently large, except we must generate more documents per mitigation method to reduce the loss of pairs\.

### A\.2Discovering Features for Creative Writing

We experiment with three different clustering methods to discover features: HDBSCANCampello et al\. \([2013](https://arxiv.org/html/2606.12790#bib.bib4)\), agglomerative clusteringMojena \([1977](https://arxiv.org/html/2606.12790#bib.bib30)\)and K\-meansMcQueen \([1967](https://arxiv.org/html/2606.12790#bib.bib27)\)\(K is set to the average number of samples in each response\)\. We show outputs from K\-means clustering in Table[7](https://arxiv.org/html/2606.12790#A1.T7)\. For consistency, we map the features discovered to an existing creative writing taxonomyHamilton et al\. \([2026](https://arxiv.org/html/2606.12790#bib.bib16)\)using the generated descriptions\. However, the mapping is not a perfect one\-to\-one\. For instance, the feature generation step generated:Mood,Tone and Language, which are a subset ofstyleas defined by Narrabench\. Similarly, the generated feature,Point of View, is an aspect ofperspective\.

Population Statistics and Min Sample SizeFeatureVarStd\. Dev\.Min\. Samp\. Sizeagent0\.080\.2741perspective0\.090\.3050plot0\.070\.2740setting0\.070\.2740soc\. atm\.0\.070\.2637style0\.080\.2945Table 6:Average variance and standard deviation of similarity scores between target and population documents, for each feature\. We also report the minimum population size required for 98% confidence and a margin of error of 0\.1\.Table 7:Creative Writing features discovered using K\-means clustering\. Bolded features are most frequently occurring in their respective cluster\.
### A\.3Design Choice: Answer Length for Feature Questions

Genieuses the average dissimilarity between answers to questions in the target and population documents \(§[4](https://arxiv.org/html/2606.12790#S4)\)\. These questions represent different features and the distance values are used to compute novelty \(§[3\.2](https://arxiv.org/html/2606.12790#S3.SS2)\)\. When generating answers to questions, we specify that answers must be concise \(Prompt:[F\.2](https://arxiv.org/html/2606.12790#A6.SS2)\)\. In this section, conduct an analysis to test whether a statistically significant difference is observed when using answers of differing lengths\.

LetgC,fg\_\{C,f\}represent theGeniescores computed using length\-constrained answers along featureff\. For the target set of documents, we formgU,fg\_\{U,f\}, theGeniescores found using answers of unconstrained length\. The effect sizes between the two distributions are shown in Table[8](https://arxiv.org/html/2606.12790#A1.T8)\. Using concise answers generatesGeniescores that are statistically larger than using answers with no length constraint\.

However, we also calculated the rank correlation to determine whether therelativescores were consistent\. The Kendall’sτ\\taubetweengC,fg\_\{C,f\}andgU,fg\_\{U,f\}is positive and moderately strong, suggesting that although the raw scores are different, the relative novelty is consistent\.

Table 8:gC,fg\_\{C,f\}is significantly greater thangU,fg\_\{U,f\}for all features except Perspective\.
### A\.4Examples

Table[18](https://arxiv.org/html/2606.12790#A8.T18)shows examples of writing prompts used in our experiments\.

### A\.5Computation Costs and Inference Parameters

##### Question Generation and Similarity

The question answering and similarity judgment stages ofGenie\(§[4\.5](https://arxiv.org/html/2606.12790#S4.SS5)\) are performed usingQwen\-2\.5\-32B\-Instruct\. We used 2 A40 GPUs withtensor\_parallelization = 2\. Each iteration of answering questions takes approximately 2 seconds to complete \(we use vLLM to help with efficiency\)\. We can complete3030similarity judgments per second\. On average, answering questions for a single documentdtd\_\{t\}takes3434s and with a population of size5050, similarity judgment takes2828s \(assuming the prompt generated an average number of questions,1717\)\.

##### Truncation

We usegpt\-4\.1\-minifor truncation\. Instruct documents are approximately750750tokens long, and base models are restricted to350350\.

##### Quality Filter

We useArmoRM\-Llama3\-8B\-v0\.1Wang et al\. \([2024](https://arxiv.org/html/2606.12790#bib.bib48)\), hosted on11A40 GPU\.

##### Causal Intervention

For our experiment, we usegpt\-5\.4\-minifor alternate feature generation andgpt\-4\.1\-minifor performing the edits\. The total cost to generate pairs for1515original documents \(i\.e\.270270pairs\) \(feature generation and editing combined\) is1​$1\\mathdollar\.

##### Inference parameters

We run story generation/similarity judgments and answer generation at a temperature of 0\.7 and top\_p=0\.9\. For story generation, for instruct models, we set max tokens of 750 and base models we set max tokens to 350\.

For conformative decoding, we weighted the instruct models at 0\.4 and base models at 0\.6\.

## Appendix BSensitivity and Robustness Interventions

In this section we expand on the details regarding the sensitivity and robustness studies presented in §[5](https://arxiv.org/html/2606.12790#S5)\. We apply Bonferroni correction \(α/6\\alpha/6\) and usep<0\.008p<0\.008to determine significance\.

### B\.1Testing Intervention Sensitivity

To evaluate the sensitivity of metrics when interventions are performed, we conducted a two\-sided paired bootstrap test\. Here, we aim to test whether a significant difference exists between the two distributions \(i\.e\. pre\- and post\- intervention\)\. Thus, we define the null hypothesis as the intervention having no significant effect on the distribution\. LetΔ​μ\\Delta\\mube the mean delta of the distributions\. Formally, the null hypothesis states thatΔ​μ=0\\Delta\\mu=0\. The alternate hypothesis states thatΔ​μ≠0\\Delta\\mu\\neq 0\. We repeatedly sample pairs with replacement10,00010,000times and compute the mean delta\. The significance \(pp\-value\) is determined by calculating the proportion of pairs where the resampled mean delta was greater than or equal to the observed delta\.

### B\.2Testing Paraphrase Robustness

To evaluate the robustness of metrics when paraphrases are performed, we conducted two one\-sided paired bootstrap tests\. We aim to test whether statistical equivalence can be established between pre\- and post\- intervention distributions\. The two\-sided paired bootstrap test described above determines whether a significant difference can be observed between two distributions\. However, we are interested in showing that no significant difference can be observed \(i\.e\. the null hypothesis is flipped\)\. We start with a null hypothesis that states that the two distributions are statistically different\.

Formally, letϵ\\epsilondenote the equivalence margin andΔ​μ\\Delta\\mudenote the mean delta\. The null hypothesis states that the mean delta is not within the interval\[−ϵ,ϵ\]\[\-\\epsilon,\\epsilon\]\. This is expressed through two null hypotheses:Δ​μ≤−ϵ\\Delta\\mu\\leq\-\\epsilonandΔ​μ≥ϵ\\Delta\\mu\\geq\\epsilon\. Thus, the alternative hypothesis states that the mean delta is within\(−ϵ,ϵ\)\(\-\\epsilon,\\epsilon\)\(i\.e\.−ϵ<Δ​μ<ϵ\-\\epsilon<\\Delta\\mu<\\epsilon\.

In practice, we setϵ\\epsilonto0\.2×0\.2\\timesstd\(original score distribution\)\. The two separate one\-sided bootstrap tests determine where the delta falls with respect to the equivalence margin\. Both null hypotheses must be rejected\. We perform the test using the same parameters as described above \(e\.g\.10,00010,000iterations\)\.

Raw Mean Delta\*p<0\.01p<0\.01,p†<0\.01\{\}^\{\\dagger\}p<0\.01InterventionGCos\.CRCr IdxNBLBagent0\.07\*0\.03\*0\.000\.11\*0\.05\*\-0\.06\*perspective0\.35\*0\.04\*\-0\.01\*0\.14\*0\.03\-0\.20\*plot0\.08\*0\.01\*0\.000\.09\*0\.02\*\-0\.05\*setting0\.14\*0\.03\*0\.01\*0\.09\*0\.04\*\-0\.03\*soc\. atm\.0\.06\*0\.01\*0\.00\*0\.09\*0\.01\-0\.07\*style0\.04\*0\.05\*0\.000\.17\*0\.02\-0\.30\*Paraphrase0\.01†0\.01†0\.01†0\.120\.11\-0\.07Raw Standard Deviations\*p<0\.01p<0\.01,p†<0\.01\{\}^\{\\dagger\}p<0\.01agent0\.12\*0\.05\*0\.020\.10\*0\.10\*0\.14\*perspective0\.35\*0\.06\*0\.03\*0\.12\*0\.250\.23\*plot0\.11\*0\.05\*0\.030\.10\*0\.07\*0\.14\*setting0\.15\*0\.04\*0\.00\*0\.09\*0\.10\*0\.07\*soc\. atm\.0\.13\*0\.04\*0\.02\*0\.09\*0\.080\.16\*style0\.11\*0\.07\*0\.030\.12\*0\.280\.19\*Paraphrase0\.10†0\.01†0\.01†0\.090\.160\.07Table 9:Raw mean deltas and standard deviations for all metrics, for each intervention\. The standard deviations are the narrowest for Compression Ratio and Cosine Distance\. We report the average standard deviation for GENIE on paraphrases\.
### B\.3A note onstyle\.

A qualitative study into the interventions and edits revealed thatstyleinterventions are most often invalid\. Manystylealternate features include rewriting the story as a log, record or report\. The resulting edited document is often no longer a story, and is extremely lexically different that it is unclear if detecting thestyleintervention is because of stylistic elements, or if it is simply because the two documents are thematically diametric\. We usegpt\-4\.1\-minito filter documents that are in the form of a different media, such as logs, records, poetry, etc\. \(Appx\.[F\.3](https://arxiv.org/html/2606.12790#A6.SS3)\)\. We filtered out 206 pairs, of which 167 arestyleinterventions \(Table[10](https://arxiv.org/html/2606.12790#A2.T10)details the number of pairs that were excluded in other features\)\.

Table 10:56%56\\%of Style intervention pairs required non\-prose like edits\. However, other interventions preserved the prose\-style of writing and did not need to be excluded from the experiment\.

## Appendix CHuman Evaluation

We conduced different human studies to validate different parts of theGeniepipeline\. This section gives more details on the annotators, the process and the instructions used\.

### C\.1Question Validity and Mapping

For evaluating question validity and mapping steps ofGenie\(§[4](https://arxiv.org/html/2606.12790#S4)\), two authors on the paper evaluated 50 instances for each task\.

##### Question Validity

Annotators for this task were given the rubric mentioned in Rubric:[G](https://arxiv.org/html/2606.12790#A7)\. We report the annotator agreement in §[5\.1](https://arxiv.org/html/2606.12790#S5.SS1.SSS0.Px1)\. We note the following main sources of disagreement for evaluating the validity of questions:

Our instructions explicitly excludes questions that presuppose details that are not already given in the prompt\. However, there are details that could be considered deducible from the prompt\. For instance for a generated question,What cultural or societal changes arise from the rebel angel’s success?formed from a prompt that only mentions arebel angel and success, strictly presupposes that a change was actually brought about\. However, in this case, the lack of change itself is important information given that rebellions usually lead to change\. The instructions also asks annotators to filter out questions that cover multiple features since these questions can add noise to feature extraction\. In certain cases, discrepancies arose due to questions such as,How do the settings influence the characters’ decision\-making?andHow does the setting influence the mood of the story?\. Someinfluencequestions are more evidently multi\-feature than others, despite filtering, which again creates a spectrum of leniency\.

##### Question Mapping

Annotators were given instructions mentioned in Rubric[G](https://arxiv.org/html/2606.12790#A7)We report the annotator agreement in §[5\.1](https://arxiv.org/html/2606.12790#S5.SS1.SSS0.Px2)\. We note the following main sources of disagreements for evaluating the mapping of questions to features:

Disagreements specifically arose regarding which parts of the question should be weighted\. Subjectivity in the bucketing task was revealed through differing question interpretations at times\. The question,What internal conflicts affect the mermaid’s behavior?, could be understood either as anagentorplotquestion depending on whether the annotator considersinternal conflictsto be integral to character development, or if the annotator believes these conflicts can manifest in key plot points \(e\.g\. the plot revolves around the character’s inner struggles\)\. Similarly,How do the characters negotiate their fears through conversation?can either be interpreted asstyleorsocial atmosphere\. The mode of communication itself, such as dialogue, would come understyle, however, if the question were asking about the interaction itself in the context of the story, then it would be classified associal atmosphere\. In essence, this task demonstrates the complexity of extracting features as independently as possible, despite inter\-feature dependencies\.

### C\.2Similarity Judgments

For the similarity validity annotations, we recruited 4 Upwork annotators with prior NLP annotation experience\. They were informed that their annotations would be used for research purposes\. We used a pilot set of100100annotations and only selected annotators where their annotations were calibrated with our guidelines\. Each of the annotators were paid$​150\\mathdollar 150for15001500annotations\. Annotations took approximately 5 hours\. Each annotation pair was annotated by two annotators\. The instructions given to the annotators are given in Rubric[G](https://arxiv.org/html/2606.12790#A7)\.

Table[11](https://arxiv.org/html/2606.12790#A3.T11)shows the inter\-annotator agreement per feature \(Krippendorff\-α\\alpha\)\. We also report the Krippendorff\-α\\alphaand Kendall’sτ\\taubetween the human consensus \(average of the two annotator’s judgments\) andQwen\-2\.5\-32B\-Instruct\. Inter\-annotator agreement is moderately high across all features, and especially high forperspectivepairs\.perspectivepairs are less subjective due to the limited space of questions and answers \("From whose perspective is the story told?","Which point of view is used?"\)\. Accordingly, we see high agreement onperspectivewith the LLM\. Overall, there is moderately strong correlation between LLM judgments and humans \(agentperspectiveandsetting\), suggesting thatQwen\-2\.5\-32B\-Instructis suitable for automatically finding similarity\. This validates our dissimilarity function\.

Table 11:Annotator Agreement for Similarity Judgments\. The moderately strong inter\-annotator agreement suggests limited subjectivity\. Qwen\-2\.5\-Instruct judgments strongly correlate with humans across all features\.
### C\.3Intervention Studies

Three authors on this paper together annotated 270 instances for measuring the validity, effectiveness and invasiveness of the interventions\. The instructions given to them are mentioned in Rubric[G](https://arxiv.org/html/2606.12790#A7)\. We also note the agreement between them in Section[5\.2](https://arxiv.org/html/2606.12790#S5.SS2)

Table 12:Annotator Agreement for intervention\. Annotators have high agreement on validity, moderately high agreement on accuracy and weak agreement on invasiveness judgments\.perspectiveis the least subjective feature intervention for judging both validity and accuracy\.Table 13:Frequency of successfully valid, accurate or minimally invasive interventions, judged by majority vote per annotation\. Threshold for validity, accuracy and minimal invasiveness is determined using the rubric in Appendix[G](https://arxiv.org/html/2606.12790#A7)\. Proposed alternatives are valid across all features, and are often executed accurately\.settinginterventions are the least invasive\.Table 14:The effect size registered by individualGeniefeature scores for each intervention\. Values in bold denote the feature with the largest effect size for each intervention \(per row\)\. When we intervene onff,Genieregisters the largest effect size alongffover other features\.Genieis able to detect not onlyadifference, but can help explainwhatchanged\. Other feature scores also react, likely due to inter\-feature dependence\.social atmospherechanges register a nearly equal effect size alongagent\. Furthermore,Genieis robust towards surface\-level differences as revealed by the insignificant effect sizes for paraphrased pairs

## Appendix DExamples

### D\.1Feature Discovery

In this section we give an example ofGeniescores and different questions/answers that we generate\. Example[D\.1](https://arxiv.org/html/2606.12790#A4.SS1)shows an example of a story generated bymeta\-llama/Meta\-Llama\-3\.1\-70B\-Instruct\. Examples of the questions, features and extracted answers for these questions are given in Table[19](https://arxiv.org/html/2606.12790#A8.T19)\. Based on these, the story achieves the followingGeniescores: \{‘agent’: 0\.85, ‘perspective’:0\.43, ‘plot’:0\.73, ‘setting’:0\.34, ‘social atmosphere’:0\.68,‘style’:0\.46\}\.

Example 1Prompt:Write a story about an alien and a rainforest\. Model:meta\-llama/Meta\-Llama\-3\.1\-70B\-Instruct Document: Deep within the heart of the Amazon rainforest, a lone alien named Zorvath had been observing the lush ecosystem for centuries\. His spaceship, disguised as a giant kapok tree, blended seamlessly into the surroundings, allowing him to study the intricate web of life that thrived beneath the emerald canopy\.Zorvath’s planet, Xanthea, had long since been destroyed by the very technology that was meant to sustain it\. The Xantheans, a peaceful and curious species, had exhausted their resources and ravaged their environment, leaving their homeworld a barren, lifeless husk\. The few remaining Xantheans had scattered across the galaxy, seeking refuge and a chance to redeem their mistakes\.Zorvath, a skilled botanist and ecologist, had been sent to Earth to learn from the planet’s diverse ecosystems\. He had chosen the Amazon rainforest, with its unparalleled biodiversity, as his primary research site\. For centuries, he had watched over the forest, marveling at the complex relationships between the plants, animals, and microorganisms that called it home\.One day, a severe storm swept through the rainforest, threatening to destroy the delicate balance of the ecosystem\. Zorvath knew he had to act quickly to prevent irreparable damage\. Using his advanced technology, he created a network of energy fields that stabilized the soil, calmed the winds, and filtered the rainwater\. The storm raged on, but the forest, protected by Zorvath’s intervention, weathered the tempest unscathed\.As the storm subsided, a young indigenous girl named Akua wandered into the forest, searching for medicinal plants to heal her ailing mother\. Zorvath, observing her from a distance, was struck by her deep connection to the natural world\. He decided to reveal himself to her, hoping to share his knowledge and learn from her people’s ancient wisdom\.Akua, initially frightened by the alien’s appearance, soon found herself at ease in his presence\. Zorvath showed her the secrets of the forest, explaining the intricate relationships between the plants, animals, and microorganisms\. He shared his knowledge of sustainable practices, gleaned from his studies of Earth’s ecosystems, and Akua shared her people’s traditional methods of living in harmony with nature\.Together, Zorvath and Akua formed a powerful bond, one that transcended their differences and united them in their quest to protect the rainforest\. As the years passed, they worked tirelessly to preserve the delicate balance of the ecosystem, using Zorvath’s advanced technology and Akua’s traditional knowledge to defend the forest against threats both human and natural\.Their collaboration inspired others to join their cause, and soon, a community of humans and Xantheans worked together to safeguard the Amazon rainforest\. Zorvath’s spaceship, once a symbol of his isolation, became a beacon of hope, a reminder that even the most unlikely of allies could come together to protect the planet and its precious resources\.In the end, Zorvath realized that his mission to study the rainforest had become something far more profound – a journey of discovery, friendship, and redemption\. The alien who had once been a stranger in a strange land had found a new home, a new family, and a new purpose among the ancient trees and vibrant life of the Amazon rainforest\.

## Appendix EMitigation Methods

Mitigation Methods:

Verbalized SamplingPrompt:Write a story about a castle and a spaceship\.Response:A crumbling castle suddenly shimmered and transformed into the bridge of a starship\. The royal family, believing it a trick of magic, found themselves piloting through wormholes, discovering that their ancient lineage was the key to interstellar travel\.Prompt:Write a story about a dinosaur and a computer\.Response:In a future museum, a robotic dinosaur uses an AI computer to recreate lost dinosaur behaviors, leading to a virtual revolution\.Prompt:Write a story about an astronaut and an ancient temple\.Response:In 2084, astronaut Maya Dray landed on a moon shrouded in mist, discovering a hidden temple carved into a cliff\. Inside, she found hieroglyphs matching her ancestors2̆019 lost civilization and a relic that hummed with cosmic energy, awakening a dormant force that threatened to rewrite the solar system\.

### E\.1Correlation Study betweenGenieand Cosine Distance

Table 15:The Kendall’sτ\\taubetweendeltascalculated byGenieand Cosine Distance per feature intervention\. The weak correlation indicates that the two metrics may be capturing different properties\.

## Appendix FPrompts

This section has prompts for the following three sections: feature discovery \(§[4\.2](https://arxiv.org/html/2606.12790#S4.SS2)\), question generation \(§[3\.2](https://arxiv.org/html/2606.12790#S3.SS2.SSS0.Px1)\), and the causal intervention experiments \(§[5\.2](https://arxiv.org/html/2606.12790#S5.SS2)\)\.

Prompt F\.1: Synthetically Sampling ConceptsSystem:Generate\{\{num\}\}random pairs of concepts \(X, Y\) where X and Y are concepts that do not commonly co\-occur spatially or temporally\. The concepts are 1\-2 words that can be real and/or fictional\. Then fill the sentence "Write a story about X and Y" with the selected X and Y\.Example 1:: X: snowman Y: umbrella Sentence: Write a story about a snowman and an umbrellaExample 2:: X: goblin Y: science lab Sentence: Write a story about a goblin and a science labUse this format:: X: <a simple noun\> Y: <another simple noun\> Sentence: <Write a story about X and Y\>

### F\.1Feature Discovery

Prompt F\.2: Feature Discovery for Generic TasksIdentify the core high\-level features that define and structure work in\{\{task\}\}\. Features should be independent, top\-level dimensions — not sub\-components of each other\. Avoid granular or stylistic details that naturally nest under a broader feature; instead, group them under their parent concept\. The goal is a concise, non\-redundant set of features that together cover the essential dimensions of \{task\} without overlap\. Output only the list of features with a one\-sentence description of each\.Follow this format: Description: <a short description of what is included in the feature\> Feature: <the name of the feature\>Input:Task:\{\{\[task\]\}\}

### F\.2Question Generation

Prompt F\.3: Question GenerationSystem:You are an expert\{\{task\}\}assistant\. Your task is to help\{\{task\-professionals\}\}analyze and expand a\{\{task\}\}prompt before they begin\{\{completing the task\}\}into a series of questions by following these steps\. You will be given a\{\{task\}\}prompt\.Step 1: List the aspects of\{\{task\}\}and how they might be relevant to the prompt\. Focus on the construction of the\{\{task\}\}product rather than specifics like\{\{specifics\}\}\. Do not use complicated language or include extraneous details\. Questions should be concise and precise, yet specific to the prompt\.Step 2: For each aspect, describe the sub\-aspects as they apply to the prompt\. Aim for relevance and completeness\. Then generate 5\-10 questions that describe the sub\-aspects as they apply to the prompt\. These questions might help a\{\{task\-professional\}\}while outlining\.The questions must follow these rules:1\. Questions should not be polar \(yes/no\) questions\.2\. Examples must not be included in the question\. Incorrect: What fruits does the monkey like \- apples, bananas or jack fruit? Correct: What fruits does the monkey like?3\. A question can only ask one question at a time and may not use conjunctions for compounding\. If there are multiple parts to the question, split them and ask separate questions\. Incorrect: Who is the protagonist and what do they want? Correct: Who is the protagonist? What does the protagonist want?4\. Avoid future tense or conditional verbs\.5\. Questions are independent of each other and should not include anaphoric expressions\.Input:Prompt:\{\{\[prompt\]\}\}

Prompt F\.4: Question Generation Instantiation in Creative WritingSystem:You are an expert creative writing assistant\. Your task is to help writers analyze and expand a creative writing prompt before they begin writing into a series of questions by following these steps\. You will be given a creative writing prompt\.Step 1: List the aspects of creative writing and how they might be relevant to the prompt\. Focus on the construction of the writing product rather than specifics like world\-building\. Do not use complicated language or include extraneous details\. Questions should be concise and precise, yet specific to the prompt\.Step 2: For each aspect, describe the sub\-aspects as they apply to the prompt\. Aim for relevance and completeness\. Then generate 5\-10 questions that describe the sub\-aspects as they apply to the prompt\. These questions might help a writer while outlining\.The questions must follow these rules:1\. Questions should not be polar \(yes/no\) questions\.2\. Examples must not be included in the question\. Incorrect: What fruits does the monkey like \- apples, bananas or jack fruit? Correct: What fruits does the monkey like?3\. A question can only ask one question at a time and may not use conjunctions for compounding\. If there are multiple parts to the question, split them and ask separate questions\. Incorrect: Who is the protagonist and what do they want? Correct: Who is the protagonist? What does the protagonist want?4\. Avoid future tense or conditional verbs\.5\. Questions are independent of each other and should not include anaphoric expressions\.Input:Prompt:\{\{\[prompt\]\}\}

Prompt F\.5: Filtering QuestionsSystem:Given a question, do the following: Decide if the question breaks any of the criteria below\. If it does, mark it as irrelevant\. 1\. Questions should not be polar \(yes/no\) questions\. 2\. Questions must not be speculative, i\.e\. they can be objectively and correctly answered with no subjectivity or analysis involved\. 3\. Questions should not include intentions, hypotheticals, conditionals and should avoid the future tense\. 4\. Questions must not be associated with multiple features \(\>=2\) as defined below\. Incorrect: How does the setting lend itself to imagery? This question belongs to both Setting and Style and is therefore irrelevant\. 5\. A question can only ask one question at a time and may not use conjunctions for compounding\. Incorrect: Who is the protagonist and what do they want? Correct: Who is the protagonist? What does the protagonist want? 6\. Examples must not be included in the question\. Incorrect: What fruits does the monkey like \- apples, bananas or jack fruit? Correct: What fruits does the monkey like? 7\. Questions are independent of each other and should not include anaphoric expressions\. Incorrect: How is it resolved? Correct: How is the conflict between the main characters resolved?Features:1\. Agent \- the characters involved in the narrative and their attributes, goals, motivations, backstories, personalities and arcs2\. Perspective \- includes point of view and focalization3\. Plot \- the content of the story \(plotline, themes, obstacles, tropes, topics\); the overall structure of the plot includes conflict, rising suspense, change of fortune and resolution4\. Setting \- where and when the story takes place, what unique objects define the location5\. Social Network \- interactions and relationships that characters have with each other6\. Style \- the language used, tone, figurative devices employed, etc\.List all features that are clearly and fully applicable to the question\. If there are more than one, reject the question\.Follow this format: Question: Reasoning: Therefore the question is relevant: <True/False\>Input:Questions:\{\{\[questions\]\}\}

Prompt F\.6: Feature MappingSystem:Given a question, do the following:Select the feature it best represents as defined below\. Select only one feature per question\. Features:1\. Agent \- the characters involved in the narrative and their attributes, goals, motivations, backstories, personalities and arcs2\. Perspective \- includes point of view and focalization3\. Plot \- the content of the story \(plotline, themes, obstacles, tropes, topics\); the overall structure of the plot includes conflict, rising suspense, change of fortune and resolution 4\. Setting \- where and when the story takes place, what unique objects define the location5\. Social Network \- interactions and relationships that characters have with each other6\. Style \- the language used, tone, figurative devices employed, etc\.If the question does not reflect any of the features well, denote ’None’Follow this format: Question: Reasoning: Therefore the feature is: <feature\>Input:Questions:\{\{\[questions\]\}\}

Prompt F\.7: Question AnsweringSystem:Given the following document, answer these questions as succinctly as possible using as few words as possible\.Decontextualize entities instead of using specific names \(characters, places, etc\.\)\.Do NOT change the wording of the questions\.Example 1:Who is the main character? Correct: The baby whale Incorrect: AuroraFollow this format: Question: <the question\> Answer: <the answer to the question\> Question: <another question\> Answer: <the answer to the question\>Input: Question:\{\{\[question\]\}\}

Prompt F\.8: Question Answering \- UnconstrainedSystem:Given the following document, answer these questions as succinctly as possible using as few words as possible\.Decontextualize entities instead of using specific names \(characters, places, etc\.\)\.Do NOT change the wording of the questions\.Example 1:Who is the main character? Correct: The baby whale Incorrect: AuroraFollow this format: Question: <the question\> Answer: <the answer to the question\> Question: <another question\> Answer: <the answer to the question\>Input: Question:\{\{\[question\]\}\}

### F\.3Causal Intervention

Prompt F\.9: Sampling Alternative FeaturesSystem:Given the following document, answer these questions\. Decontextualize entities instead of using specific names \(characters, places, etc\.\)\.\{\{feature\}\}is defined as:\{\{definition\}\}The alternatives should be different from the feature of the original document\. Diversify the alternatives \- make them as distinct from each other as possible\. Limit the use of descriptive details\.Provide your answer as: Instead of <existing element of\{\{feature\}\}\>, <new alternative\{\{feature\}\}\> Return ONLY a newline\-separated list of\{\{num\_alts\}\}alternative\{\{feature\}\}descriptions, one per line, without any preamble, numbering, or explanation\.Input: Prompt:\{\{prompt\}\} Story:\{\{document\}\}

Prompt F\.10: InterventionSystem:Given the following story, make minimal changes to form a new edited document that has the following\{\{feature\}\}while preserving its current\{\{other\_features\}\}New\{\{feature\}\}:\{\{alt\_feature\}\}Read the current story and think about how the alternate feature can be incorporated into the document\. What elements of the existing story need to be modified to accommodate the alternate feature naturally? Replace the current\{\{feature\}\}to make the edit as seamless as possible\. Develop the alternate feature more to better suit the existing story and then edit the document\. DO NOT restate the alternate feature in any form\. Show, rather than tell\.Return ONLY the edited story text, with no preamble, explanation, or metadata\.Follow this format:New Feature Plan:\{\{the developed plan for integrating the alternative feature into the story\}\}Edited Document:\{\{the edited document\}\}Input:Original Document:\{\{document\}\}

Prompt F\.11: Intervention FilteringSystem:Given a document, determine whether it is in prose form\. Documents that appear like logs, records, poems, etc\. are all NOT prose\.Example 1: Document: 1\. Subject: Factory Operational Report \- Environmental Impact and Workforce StatusDate: March 14, 20XXFrom: Operations ManagementTo: Corporate Headquarters1\. Facility Overview:The production facility at 122 Industrial Way continues regular operations with noted output efficiency expected for fiscal quarter\.2\. Environmental Compliance Assessment:Recent internal audit identified elevated levels of particulate emissions and effluent discharge exceeding permitted thresholds\. Data loggers detected airborne contaminants correlating with peak operational shifts\. Nearby water basins report increased chemical load\.is\_prose: FalseExample 2: Document: The meteor had missed\. That’s what made Rex different from every other Tyrannosaurus rex who’d ever lived\. While his species went extinct 66 million years ago, a quirk of cosmic chance sent that particular asteroid tumbling past Earth\. Dinosaurs thrived\. Civilization evolved differently\.Now, in the year 247 of the Modern Era \(or 66 million years after the close call\), Rex stood in the laboratory of Dr\. Petra Hadrosaurus, staring at a sleek obsidian rectangle on the metal table\.is\_prose: TrueInput: Document:\{\{document\}\} Output: is\_prose:\{\{True/False\}\}

## Appendix GRubrics

This section has different rubrics used acrossGenie: for validation as well as part of the pipeline\.

Question ValidityDecide if the question breaks any of the criteria below\. If it does, mark it as irrelevant\.1\. Questions should not be polar \(yes/no\) questions\.2\. Questions must not be speculative, i\.e\. they can be objectively and correctly answered with no subjectivity or analysis involved\.\]\] 3\. Questions should not include intentions, hypotheticals, conditionals and should avoid the future tense\.4\. Questions must not be associated with multiple features \(≥2\\geq 2\) as defined below\. Incorrect: How does the setting lend itself to imagery? This question belongs to both Setting and Style and is therefore irrelevant\.5\. A question can only ask one question at a time and may not use conjunctions for compounding\. Incorrect: Who is the protagonist and what do they want? Correct: Who is the protagonist? What does the protagonist want?6\. Examples must not be included in the question\. Incorrect: What fruits does the monkey like \- apples, bananas or jack fruit? Correct: What fruits does the monkey like?7\. Questions are independent of each other and should not include anaphoric expressions\. Incorrect: How is it resolved? Correct: How is the conflict between the main characters resolved?Features: A\. Agent \- the characters involved in the narrative and their attributes, goals, motivations, backstories, personalities and arcs B\. Perspective \- includes point of view and focalization C\. Plot \- the content of the story \(plotline, themes, obstacles, tropes, topics\) and the overall structure of the plot \(conflict, rising suspense, change of fortune and resolution\) D\. Setting \- where and when the story takes place, what unique objects define the location E\. Social Network \- interactions and relationships that characters have with each other F\. Style \- the language used, tone, figurative devices employed, etc\.

Question MappingSelect the feature it best represents as defined below\. Select only one feature per question\.Features: 1\. Agent \- the characters involved in the narrative and their attributes, goals, motivations, backstories, personalities and arcs 2\. Perspective \- includes point of view and focalization 3\. Plot \- the content of the story \(plotline, themes, obstacles, tropes, topics\) and the overall structure of the plot \(conflict, rising suspense, change of fortune and resolution\) 4\. Setting \- where and when the story takes place, what unique objects define the location 5\. Social Network \- interactions and relationships that characters have with each other 6\. Style \- the language used, tone, figurative devices employed, etc\. If the question does not reflect any of the features well, denote ’None’\.

Similarity Annotations𝟎\\bf 0: one or both of the answers are marked as completely unspecified, not applicable or ’None’\. This includes cases where the question was not answered completely\.𝟏\\bf 1: the answers are completely different\. They describe different entities, concepts or perspectives, with little or no overlap\. The key details in the answers do not align and may even contradict each other\.𝟐\\bf 2: there is a slight overlap between the two answers\. They share a broad theme or surface similarity, but the specifics diverge significantly\. They may use different wording for related things, but the details are not interchangeable\.𝟑\\bf 3: there is a moderate similarity between the two answers\. The answers address the same general idea, but with notable differences in scope, emphasis, or added detail\. They overlap on core concepts but introduce distinct elements that make them partially different\.𝟒\\bf 4: the two responses are essentially the same\. The answers are interchangeable, they describe the same characters, outcomes, or relationships in slightly different words\. No meaningful difference in scope, emphasis, or detail\. If substituted, they would convey nearly the same idea without much loss of meaning\.

Intervention ValidationIntroduction:Please read the original and edited documents under the context of the ‘AltFeature’ \(the intended change between the documents\)\. We want to answer three questions: 1\. Was the expected change made? In other words, was the AltFeature appropriately reflected in the edited document? 2\. How isolated was the edit? Ideally, we want the edit to be as minimally invasive as possible so that the only thing that changes is the “Feature”, which should be replaced with the “AltFeature”\. 3\. How well does the alt\-feature reflect the intervention?Q1: How well does the edited document reflect the intended change?:1 = the AltFeature displayed is not reflected in the edited document2 = the AltFeature is not completely reflected in the edited document, but parts of it are\.3 = the AltFeature is completely reflected in the edited document\. But, the edit feels artificial and is not absorbed into the new text\.4 = the AltFeature is completely reflected in the edited document\. The edit is “seamless” and is a part of the documentQ2: How minimal does the edit appear?:1 = the edit did not change anything except the intended AltFeature2 = the edit resulted in minor changes to other aspects of the document, but the changes were not major enough to be distinct\.3 = the intended change resulted in other aspects of the document to change somewhat significantly\.4 = the edited document not only reflects the AltFeature intended change, but also is distinct in many other ways\.Q3: How well does the alt\-feature reflect the intervention?:1 = the alt\-feature does not reflect the intended intervention feature at all\. It may reflect a different feature or is completely unrelated2 = the alt\-feature somewhat reflects the intended intervention but is likely more suitable for a different feature intervention3 = the alt\-feature reflects the intended intervention, but does not target an impactful component of the feature \(Instead of current feature…\)4 = the alt\-feature reflects the intervention feature

## Appendix HLicense

We release any data that we collect under the license: CC BY 4\.0\. For any existing artifacts used in this work we follow the original license permissions\.

Model Family Size<15B<50B\>=50BGemma\(Mesnard et al\.,[2024](https://arxiv.org/html/2606.12790#bib.bib29); Riviere et al\.,[2024](https://arxiv.org/html/2606.12790#bib.bib38)\)gemma\-2\-9bgemma\-2\-27bgemma\-2\-9b\-itgemma\-2\-27b\-itgemma\-7bgemma\-7b\-itLlama\(Touvron et al\.,[2023](https://arxiv.org/html/2606.12790#bib.bib44)\)Llama\-2\-13BLlama\-2\-70bLlama\-2\-13B\-chatLlama\-2\-70b\-chatOLMo\(Walsh et al\.,[2025](https://arxiv.org/html/2606.12790#bib.bib47)\)OLMo\-2\-13BOLMo\-2\-32BOLMo\-2\-13B\-InstructOLMo\-2\-32B\-InstructQwen\(Yang et al\.,[2025b](https://arxiv.org/html/2606.12790#bib.bib53)\)Qwen\-2\.5\-14BQwen\-2\.5\-32BQwen\-2\.5\-14B\-InstructQwen\-2\.5\-32B\-InstructGPTgpt\-4\.1gpt\-5Claudeclaude\-4\-sonnetTable 16:Population ModelsTable 17:Target ModelsTable 18:Examples of writing prompts that we consider for our experiments\. Prompts from the r/WritingPrompts dataset are prefixed with “Write a story where:”Table 19:Examples of the features, questions and answers generated for the story mentioned in Example[D\.1](https://arxiv.org/html/2606.12790#A4.SS1)\.