Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study

# Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study
Source: [https://arxiv.org/html/2607.26221](https://arxiv.org/html/2607.26221)
Angshu Nirmegh BiswasDepartment of Computer Science and EngineeringBRAC Universityangshu\.nirmegh\.biswas@g\.bracu\.ac\.bdTaspiha TabassumDepartment of Computer Science and EngineeringBRAC Universitytaspiha\.tabassum@g\.bracu\.ac\.bdAfia Abida ShohidDepartment of Computer Science and EngineeringBRAC Universityafia\.abida\.shohid@g\.bracu\.ac\.bd Razia Marzan MouDepartment of Computer Science and EngineeringBRAC Universityrazia\.marzan\.mou@g\.bracu\.ac\.bdAyeeshah Akter EshaDepartment of Computer Science and EngineeringBRAC Universityayeeshah\.akter\.esha@g\.bracu\.ac\.bd SupervisorDr\. Farig Yousuf SadequeAssociate ProfessorDepartment of Computer Science and EngineeringBRAC UniversityCo\-supervisorAnika AhmedLecturerDepartment of Computer Science and EngineeringBRAC University

###### Abstract

With the advancement of AI technologies, Generative AI \(GenAI\) and human written text have become nearly indistinguishable\. Additionally, the global standardization of AI chatbots made academic malpractice more frequent\. Furthermore, existing research indicates GenAI poems are the most difficult to distinguish even without any modification thus, GenAI poems are naturally deemed “human\-like” by modern detectors\. However, the objectivity of such dissertations needs to be verified against modern detection tools but the subjectivity of poetry and the “black\-box” nature of the modern LLMs \(Large Language Models\) architectures made verification of such work quite complicated\. Hence, the main objective of the research is to deduce the attributes of English poetry that contribute classification and misclassification of both human and AI poems and provide corroborating or contradicting evidence to the poetry distinguishability claim\. For such characterizations, we propose a Zero\-shot detection pipeline with a dataset consisting of both human and AI poems to verify the distinguishability of human and AI creation and extract the aforementioned crucial attributes for accurate classification\. Extraction of such attributes provides benefits in two ways: firstly, it reduces the margin of training needed as only the poems based on misclassifying attributes need to be trained and fine tuned and finally provides a critical insight to the GenAI detection dilemma to strengthen the modern detection pipelines\.

## IIntroduction

The use of GenAI has become very popular in academic writing and other fields resulting in academic malpractice and plagiarism which is especially true for the domain of the poetry\. Poetry usually transcends the boundary of the grammatical structure and introduces ambiguity in forms of artistic expression\. Such form of creativity blurs the line between human creativity and machine generated poems\. The modern large language models show a strong ability in generating creative outputs, particularly poetry, by the use of metaphors, figurative language and stylistic variation which are traditionally associated with human creativity\. Hence, surface detection metrics based on perplexity, burstiness, logarithmic probability \(logprobs\)\[[1](https://arxiv.org/html/2607.26221#bib.bib1)\]and other forms of statistical analysis may struggle to accurately classify AI generation from human poems\. Therefore, GenAI poetry detection requires extra depth, nuances and most importantly steady markers which we dubbed to be “Human\-like” attributes to overcome the unreliable nature of statistics based classification\. Moreover, the current detection tools are mostly trade secrets, which causes limited access for collaborative improvement\. This is why, it is essential to develop open access systems that can not only identify the AI generated content but also provide room for improvement as collaboration will allow further human\-like attributes and enhance the detection pipeline further by mending the underlying limitations\.

## IIRelated Works

Difficulty of GenAI poetry classification is stated both through the naked eyes\[[3](https://arxiv.org/html/2607.26221#bib.bib3)\]and the automated classification systems\[[2](https://arxiv.org/html/2607.26221#bib.bib2)\]\. This research aims to validate the aforementioned claims under one experimental setup and deduce how difficult it is to distinguish GenAI poems from human authored ones\. Additionally, in poetry classification, MERMAID\[[5](https://arxiv.org/html/2607.26221#bib.bib5)\]and Chen et al\.\[[4](https://arxiv.org/html/2607.26221#bib.bib4)\]highlighted the impacts of lexical diversity of GenAI poems and the usage of Metaphors as masking agents to humanize GenAI creation\. Firstly, Chen et al\. explains creative depth in human creation where human contents outrank models such as, GPT\-2, GPT\-Neo, LLaMA\-2 and LLaMA\-3 in lexical diversity and rhyming patterns\. Secondly, per Chen et al\., guided and structured based generation showed LLMs’ tendency to memorize and produce even less diverse output\. However, MERMAID demonstrated using literary contents such as metaphors, the rigidness of GenAI contents can be broken\. Therefore, just as MERMAID, this research aims to explore what other literary attributes blurs the line between human and machine generated content and answer whether structured generation \(few\-shot and style based prompting\) can incorporate such literary attributes in GenAI content and moreover, what are the impacts of such attributes in the classification process as of whole\. For this purpose, GPT OSS\-120b\[[6](https://arxiv.org/html/2607.26221#bib.bib6)\], Qwen 3\-32B\[[8](https://arxiv.org/html/2607.26221#bib.bib8)\], Llama 3\.3\-70B\[[7](https://arxiv.org/html/2607.26221#bib.bib7)\]were selected to generate poems in a stylized and human few\-shot setting\. Finally, the combination both of human samples, generated creation of the aforementioned models were passed on to classify through Gemma 4\-31B\[[13](https://arxiv.org/html/2607.26221#bib.bib13)\]\(in a Zero\-Shot environment\) and to human survey participants to affirm the claims of RAID benchmark and Porter and Machery and record the “Humanizing” and the “Non\-humanizaing” attributes to strengthen the classification\.

## IIIMethodology Overview

The research methodology is divided into three branches: dataset creation, classification, and analysis\.

DATACLASSIFICATIONANALYSISDataset Creation∼\\sim5,970 poems⋅\\cdot4 LLM sourcesDataset PreprocessingCleaning⋅\\cdotNormalization⋅\\cdotSplittingClassificationMethodsTraditionalDetectorsGemma 4\-31BHumanSurveyTraditional DetectorsClassify human vs AIGemma 4\-31BZero\-shot pipelineClassify human vs AIHuman Survey320 responses16 evaluatorsClassifyRecord predictionsClassifyRecord predictionsClassifyRecord criterion\-wise predictions and pairwise agreementPrediction ComparisonInter\-traditionaldetectorsDefining AttributesUse Gemma predictionsto identify AI vs human markersSurvey Rating AnalysisIdentify consistent distinguishing criteriaDocument Results & AnalyzeCompare, report, visualizeKey InsightsAttributes driving detection & evasionDataset creationPreprocessingLLM classifierHuman/outputTraditionalDecisionResultsInsights

Figure 1:Overview of the proposed methodology for detecting AI\-generated poetry\.The elaboration of each branch is described in the following subsections\.

### III\-ADataset Generation Pipeline in Detail

![Refer to caption](https://arxiv.org/html/2607.26221v1/images/Data_gen.png)Figure 2:Data generation pipelineThe workflow shown in Fig\.[2](https://arxiv.org/html/2607.26221#S3.F2)represents how data samples are generated by LLMs based on the human sample containing human\-written poems\. In this section, the generation pipeline is described in detail\.

#### III\-A1Human Sample Collection

Human\-written poems were gathered from three sources: poetry posted publicly on social media platforms, personal archives shared directly by authors after permission was requested, and open\-access literary websites where authors publish their poems\. We approached the authors individually for permission to work with their poems, and 1,513 human poems were collected through this process, including 11 authors and diverse writing styles\.

#### III\-A2Initialization and LLM Generation Loop

For poem generation, three LLMs were selected based on novelty: GPT\-OSS\-120B, Llama 3\.3\-70B, and Qwen 3\-32B\. Before invoking the LLMs, each incomplete generation request was packaged as a job\. This package included a predetermined version of the Universal Style Guide and a randomly selected theme a global theme array containing 20 traditional poetry themes\. The job also included two few\-shot examples based on the author style being mimicked\. The purpose of the few\-shot style examples was to help the models understand and learn the author’s style without memorizing or copying the work verbatim\. This job packing framework is much lighter than Wang et al\.’s\[[15](https://arxiv.org/html/2607.26221#bib.bib15)\]image based poetry generation ableit at the cost less sophisticated generation\. Finally, the jobs were shuffled and sent to the LLMs for generation\.

##### Universal Style Guide

- •Titling:A meaningful, original title must precede the generated text\.
- •Linguistic Constraints:Outputs are strictly limited to English, and the use of em dashes \(—\) is explicitly prohibited\.
- •Originality:The model must mimic the stylistic voice of the provided examples without plagiarizing content, ensuring a novel generation based on the assigned theme\.

##### Global Thematic Pool

T=\{Love,Romanticism,Heartbreak,…,Wonder,Resilience\}T=\\\{\\text\{Love\},\\text\{Romanticism\},\\text\{Heartbreak\},\\dots,\\text\{Wonder\},\\text\{Resilience\}\\\}\(1\)Here,TTrepresents the global theme array, which was later packed as part of each job\.

##### Dynamic Few\-Shot Construction

Sa=RandomSample​\(Da,min⁡\(n,\|Da\|\)\)S\_\{a\}=\\text\{RandomSample\}\(D\_\{a\},\\min\(n,\|D\_\{a\}\|\)\)\(2\)PF​S​\(a\)=Ip​r​e​f​i​x⊕⨁s∈Sa\(Title​\(s\)⊕Poem​\(s\)\)P\_\{FS\}\(a\)=I\_\{prefix\}\\oplus\\bigoplus\_\{s\\in S\_\{a\}\}\\Big\(\\text\{Title\}\(s\)\\oplus\\text\{Poem\}\(s\)\\Big\)\(3\)Here,SaS\_\{a\}is the subset of sampled few\-shot examples for target authoraa\.DaD\_\{a\}is the subset of the human dataset containing only works authored byaa, and\|Da\|\|D\_\{a\}\|is the total number of available works for that author\.nnis the target number of few\-shot examples to include, which was 2 in this study\.PF​S​\(a\)P\_\{FS\}\(a\)is the final constructed few\-shot prompt for authoraa\(See Example:[B](https://arxiv.org/html/2607.26221#A2)\)\.Ip​r​e​f​i​xI\_\{prefix\}represents the initial system instruction block, or Universal Style Guide\.ssis an individual sampled record from whichTitle​\(s\)\\text\{Title\}\(s\)andPoem​\(s\)\\text\{Poem\}\(s\)are extracted as strings\.

##### Job Queue Creation

Algorithm 1: Job Queue Formulation with Resume Logic

0:Human dataset

DD, model set

MM, theme set

TT, and completed jobs

CC
0:Randomized pending task queue

JJ
1:

J←∅J\\leftarrow\\emptyset
2:foreach

\(i​n​d​e​x,e​n​t​r​y\)∈D\(index,entry\)\\in Ddo

3:

a​u​t​h​o​r←e​n​t​r​y\.Authorauthor\\leftarrow entry\.\\text\{Author\}
4:foreach

m​o​d​e​l∈Mmodel\\in Mdo

5:if

\(i​n​d​e​x,m​o​d​e​l\)∉C\(index,model\)\\notin Cthen

6:

t​h​e​m​e←RandomSample​\(T\)theme\\leftarrow\\text\{RandomSample\}\(T\)
7:

J←J∪\{\(m​o​d​e​l,a​u​t​h​o​r,t​h​e​m​e,i​n​d​e​x\)\}J\\leftarrow J\\cup\\\{\(model,author,theme,index\)\\\}
8:endif

9:endfor

10:endfor

11:return

Shuffle​\(J\)\\text\{Shuffle\}\(J\)

From the incomplete generation tasks, queueJJis created\. The ledger of completed tasks,CC, and the valuei​n​d​e​xindexare kept to track progress and make the overall process crash\-consistent during API calls\.

##### LLM Invocation

Or​a​w=fm​\(PF​S​\(a\)⊕Pu​s​e​r​\(t\)\)O\_\{raw\}=f\_\{m\}\(P\_\{FS\}\(a\)\\oplus P\_\{user\}\(t\)\)\(4\)Here,Or​a​wO\_\{raw\}is the raw generated output string returned by the language model\.fmf\_\{m\}is the function representing the LLM, wherem∈Mm\\in M\.PF​S​\(a\)P\_\{FS\}\(a\)is the complete style guide constructed in Eq\. \([3](https://arxiv.org/html/2607.26221#S3.E3)\)\.Pu​s​e​r​\(t\)P\_\{user\}\(t\)is the explicit user prompt instructing the model to generate a text based on the designated themet∈Tt\\in T\.

### III\-BDataset Overview and Preprocessing

The final dataset contains 5,970 poems, including 4,539 AI\-generated poems and 1,513 human\-authored poems which is signifiacntly more than the POEMetric corpus\[[14](https://arxiv.org/html/2607.26221#bib.bib14)\]\. AI poems were generated using Llama 3\.3\-70B, GPT\-OSS\-120B, and Qwen 3\-32B\. All poems were generated under the few\-shot prompt setting\. Each model generated 1,513 poems, and each human poem served as a style reference during few\-shot generation\. From a pool of 20 universal themes, each theme was assigned randomly\.

#### III\-B1Data Preprocessing

A cleaning pipeline was applied to preprocess raw outputs from AI\-generated poems\. First, chain\-of\-thought blocks enclosed in<think\>were removed in cases where the model failed to close the tag after reaching token limits\. Second, leftover generation code fences and language identifiers were removed\. Third, phrases such as “Here is a poem” and explicitTitle:orTheme:headers were stripped\. Typographic and symbolic noise, including em dashes, Unicode dash variants, and double\-hyphen variants, was removed from line endings and replaced with spaces elsewhere\. Standard punctuation was retained, while mathematical operators, brackets, and special symbols were removed\. Finally, triple dots and ellipses were normalized\. Because this study focuses on English poems, non\-English words generated by Qwen inside English poems were filtered out\. Whitespace was standardized, and consecutive blank lines were reduced to one empty line between stanzas\. From the 4,539 AI samples, 82 poems were removed during cleaning because of poor generation quality\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/images/Cleaning.png)Figure 3:Data cleaning visualization
#### III\-B2Feature Extraction

To compare the differences between AI\-generated and human poems, vocabulary\- and phrase\-based features were extracted from the dataset\. The analysis included total word count, unique words, type\-token ratio \(TTR\), most frequent words, bigrams, trigrams, semantic consistency, prompt repetition, vocabulary overlap, and TF\-IDF\. The summary of the Feature Extraction methods can be found at Table[I](https://arxiv.org/html/2607.26221#S3.T1)\. These features were used to determine whether AI\-generated poems rely on the semantics produced by human authors\.

TABLE I:Extracted vocabulary and n\-gram features
#### III\-B3Theme\-Based Dataset and Distribution

From the AI\-generated poems across 20 themes, theme analysis was performed to examine whether vocabulary patterns change when poems are grouped by theme\.

TABLE II:Poem count per theme and model![Refer to caption](https://arxiv.org/html/2607.26221v1/images/theme_percentage_distribution.png)Figure 4:Theme percentage distribution across all LLM invocation callsThe theme distribution is almost balanced across the models, with the highest difference between the themes Heartbreak and Mortality being approximately 1% \(Fig\.[4](https://arxiv.org/html/2607.26221#S3.F4)\) and Table[II](https://arxiv.org/html/2607.26221#S3.T2)displays theme distribution per model is also near equal\.

### III\-CHuman Classification via Survey

#### III\-C1Custom Survey System

We built a web application to collect survey responses from humans\. This website served as a platform for collecting human evaluations to understand perceptions of poems in our custom dataset\. The primary goal was to gather detailed judgments from users who analyzed poems according to different metrics and decided whether each poem appeared to be AI\-generated or human\-written\. Every participant was presented with a set of twenty poems and asked to evaluate them using a consistent and structured approach\.

The evaluation process used five criteria: clarity of expression, use of literary devices, punctuation and spacing patterns, grammar and spelling accuracy, and overall originality\. For each criterion, participants rated how likely they thought the poem was AI\-generated using a Likert scale ranging from “Very Unlikely” to “Very Likely”\. The platform also offered an optional comment section where evaluators could describe specific elements that influenced their decisions\.

The website was designed for both user convenience and research integrity\. Participants began by providing an email address, which acted as a unique identifier and helped track progress\. A progress bar displayed how many poems the user had evaluated out of twenty\. The system saved progress so that participants could resume the survey later\. The system also assigned poems in a way that maintained fair distribution across users while preserving research validity\.

### III\-DGemma 4 Zero\-Shot Classification

Gemma 4\-31B was used as a filter to identify patterns exhibited by GenAI poems\. Previous research has shown that GenAI poetry can be difficult to distinguish from human poetry and may even be preferred over human\-written poems\. Therefore, to extract telltale AI patterns, Gemma 4 was used in a zero\-shot setting, meaning it had no prior knowledge or training on our dataset of 5,970 samples\. This method also helped identify which machine\-generated attributes appeared human\-like\. These machine\-like and human\-like attributes produced hard positive results, which we call defining attributes, and hard negative results, which are attributes that force misclassification\.

#### III\-D1Ensuring the Zero\-Shot Setting

To ensure the zero\-shot setting, Gemma 4 was queried with a single random line from human poems and asked to return the next line\. Gemma was queried over 100 human samples from 11 human authors\. If Gemma returned the next poetry line correctly, that author’s work would be considered contaminated and discarded from the human dataset\.

#### III\-D2Classification

The classification process required the dataset to be shuffled to reduce unpredictable ordering biases\. Each sample was queried to be classified using the same five criteria and Likert scale as the survey, with three additional responses: classification label, confidence score from 0–100, and reasoning factor\. The reasoning factor asked which of the five criteria most influenced the label, or requested another factor if none of the five applied\.

Algorithm 2: Gemma 4 Detection Classification Pipeline

Input:Raw datasetℱin\\mathcal\{F\}\_\{\\text\{in\}\}, checkpointℱout\\mathcal\{F\}\_\{\\text\{out\}\}, and criteria prompt𝒮prompt\\mathcal\{S\}\_\{\\text\{prompt\}\} Output:Evaluated DataFrame continuously committed toℱout\\mathcal\{F\}\_\{\\text\{out\}\}

1:Load

𝒟​ℱ\\mathcal\{DF\}from

ℱout\\mathcal\{F\}\_\{\\text\{out\}\}if it exists; otherwise load

ℱin\\mathcal\{F\}\_\{\\text\{in\}\}
2:Initialize missing detection labels, confidence, reasoning, five metric columns, and processed flags

3:Cast

𝒟​ℱ​\[Processed\]\\mathcal\{DF\}\[\\text\{Processed\}\]to Boolean and set metric columns to object datatypes

4:foreach row index

iiand record

RRin

𝒟​ℱ\\mathcal\{DF\}do

5:if

R​\[Processed\]R\[\\text\{Processed\}\]is truethen

6:continue to next record

7:endif

8:

p​o​e​m​\_​t​e​x​t←R​\[poem\]poem\\\_text\\leftarrow R\[\\text\{poem\}\]
9:if

p​o​e​m​\_​t​e​x​tpoem\\\_textis null or emptythen

10:mark

RRprocessed and continue

11:endif

12:

𝒮flag←False\\mathcal\{S\}\_\{\\text\{flag\}\}\\leftarrow\\text\{False\}
13:while

𝒮flag\\mathcal\{S\}\_\{\\text\{flag\}\}is falsedo

14:ifGemma API returns valid

r​a​w​\_​c​o​n​t​e​n​traw\\\_contentthen

15:

𝒥←Extract\_JSON\_From\_Text​\(r​a​w​\_​c​o​n​t​e​n​t\)\\mathcal\{J\}\\leftarrow\\text\{Extract\\\_JSON\\\_From\\\_Text\}\(raw\\\_content\)
16:if

𝒥\\mathcal\{J\}is a valid dictionarythen

17:Copy label, confidence, reasoning, five metric scores, and processed flag from

𝒥\\mathcal\{J\}to

RR
18:else

19:Store raw output in

R​\[Reasoning\]R\[\\text\{Reasoning\}\]and mark

RRprocessed

20:endif

21:else

22:Store empty\-response error in

R​\[Reasoning\]R\[\\text\{Reasoning\}\]and mark

RRprocessed

23:endif

24:Save

𝒟​ℱ\\mathcal\{DF\}to

ℱout\\mathcal\{F\}\_\{\\text\{out\}\}, set

𝒮flag\\mathcal\{S\}\_\{\\text\{flag\}\}true, and sleep for 2 seconds

25:On network exception, log the error and sleep for 60 seconds

26:endwhile

27:endfor

The𝒮f​l​a​g\\mathcal\{S\}\_\{flag\}flag andR​\[Processed\]R\[\\text\{Processed\}\]record state were used to prevent API token overuse and make the system crash\-consistent\.

### III\-ETraditional Detector Classification

Apart from Gemma 4, four additional detectors were used to evaluate the dataset as comparison metrics: Log\-Likelihood\[[11](https://arxiv.org/html/2607.26221#bib.bib11)\], Log\-Rank Ratio \(LRR\)\[[12](https://arxiv.org/html/2607.26221#bib.bib12)\], Binoculars\[[10](https://arxiv.org/html/2607.26221#bib.bib10)\], and Fast\-DetectGPT\[[9](https://arxiv.org/html/2607.26221#bib.bib9)\]\. These detectors are pretrained and can efficiently classify unaltered GenAI content\. Testing our dataset against these models provides insight into Gemma 4’s classification capability and the evasiveness of the dataset\.

## IVResults

The results are organized around lexical diversity, n\-gram overlap, survey\-based human evaluation, Gemma classification, traditional detector performance, and qualitative attributes of both GenAI and human\-authored poems\.

### IV\-AWord Diversity Analysis

GPT\-OSS\-120B, Llama\-3\.3\-70B, and Qwen\-3\-32B were compared with human samples using the methods explained in[I](https://arxiv.org/html/2607.26221#S3.T1)\. The following subsections elaborate on the word diversity analysis\.

#### IV\-A1Type\-Token Ratio and TF\-IDF Analysis

TTR was calculated for each corpus by dividing the number of unique word types by the total number of word tokens after removing stopwords\. The human corpus achieved the highest TTR, indicating the richest vocabulary relative to corpus size\. Among the AI models, Qwen achieved the highest TTR, while GPT and Llama showed lower word variation and higher repetition\.

TABLE III:Total words, unique words, and TTR by sourceThe Table[III](https://arxiv.org/html/2607.26221#S4.T3)explains that individual human poets used more varied vocabulary, while AI\-generated corpora shared more overlapping lexical patterns resembling the findings of Chen et al\.\[[4](https://arxiv.org/html/2607.26221#bib.bib4)\]\.

#### IV\-A2Model\-Specific Vocabulary Tendencie

The human corpus is mixed with different types of vocabulary, while the three AI models show more concentrated tendencies\. GPT focused on atmospheric vocabulary, Llama focused on emotional abstractions, and Qwen produced variations closest to human vocabulary\. Thus, human poetry remains more diverse\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/prism-uploads/p06_img01.png)Figure 5:Word frequency distribution visualization
#### IV\-A3Bigram and Trigram Feature Extraction

The analysis also shows phrase\-combination patterns\. Table[IV](https://arxiv.org/html/2607.26221#S4.T4)shows unique bigram and trigram counts per source\.

TABLE IV:Unique bigram and trigram counts by sourceTABLE V:Top 5 N\-grams per Corpus\.Human sample bigrams have colloquial phrases \(Table[V](https://arxiv.org/html/2607.26221#S4.T5)\) that are mostly absent in AI poems\. The three models use distinctive phrases: GPT uses atmospheric phrases, Llama uses emotional abstractions, and Qwen has more concrete and distinctive phrases\. Such patterns can lead to misclassification of Qwen’s poems as human\-made\. Finally, at the trigram level, repetition is rare\.

#### IV\-A4Theme\-Based Analysis: Same Theme vs\. Different Theme

Vocabulary patterns are influenced by theme frequency and were examined by dividing the poems into same\-theme and different\-theme groups\. The same\-theme group contained frequently generated words, while the different\-theme group contained less frequent words\. Although the theme distribution among the three models was mostly even, the same\-theme group showed a limited set of recurring terms, whereas the different\-theme group showed more vocabulary richness\. The contrast clarified that same\-theme poems had lexical repetition of poetic expressions, while different\-theme poems introduced more varied word choices into the corpus\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/prism-uploads/p09_img01.png)Figure 6:Theme\-wise word cloud for same and different theme pairs
#### IV\-A5Semantic Consistency Analysis

Semantic consistency was measured by counting cases where a poem was assigned to a specific theme\. This identifies whether the model stays focused on the assigned theme or followed the human sample semantics exactly, causing theme leakage\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/prism-uploads/cell_112_output_1_image_35.png)Figure 7:Semantic consistency analysis visualizationThe results indicate theme leakage, where models \(Especially Llama\) became more biased toward few\-shot examples than toward the assigned theme\.

#### IV\-A6Prompt Repetition

Prompt repetition shows direct repetition of assigned theme words such as beauty, love, and joy etc\. in the generated poems\. This was calculated by counting the frequency of assigned theme words appearing in poems of that theme\. It measures thematic adherence where the model relies on the prompt word as a surface marker rather than developing the theme through poetic style\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/prism-uploads/cell_107_output_1_image_33.png)Figure 8:Theme word repetition analysis visualizationThe results show that themes such as Beauty, Desire, Faith, Love, and Joy appeared quite often in generated poem bodies, but not enough to offset the few\-shot bias\.

### IV\-BN\-Gram Vocabulary Overlap Analysis: Human vs\. AI

The combined AI dataset shared 7514 unique words with the human corpus, representing 55\.36% of the human vocabulary and 62\.99% of the combined AI vocabulary\. However, overlap dropped sharply at the bigram and trigram levels, showing that AI models may replicate human\-like word choices while still producing distinct phrase combinations\.

TABLE VI:Full overlap metrics for Human versus AI combinedAcross GPT, Llama, and Qwen, bigram and trigram overlap values remained low, indicating that phrase combinations are both model\-specific and human\-specific\. Qwen showed stronger lexical diversity, which aligns with its higher evasion tendency in later classification results\. Finally, for the vocabulary overlap per model, the Table[VII](https://arxiv.org/html/2607.26221#S4.T7)shows, at the word level, GPT has the highest share of its own vocabulary covered in human samples at 76\.72%, which is nearly three quarters of GPT’s unique words\. Llama follows at 74\.52%, while Qwen reaches 68\.31%\. However, Qwen covers the largest share of human vocabulary at 42\.68%, compared with GPT at 38\.54% and Llama at 32\.22%, which is coherent with the TTR analysis\.

TABLE VII:Vocabulary and n\-gram overlap between each AI model and HumanAcross the three models, bigram and trigram values are quite low, making it clear that phrase combinations are model\-specific and human\-specific\.

### IV\-CSurvey Result Analysis

The experimental dataset includes a total of 320 survey evaluations collected across an allocated corpus of 180 unique poems\. The corpus was divided into 9 distinct survey slots consisting of 20 poems each\. The poems were evaluated independently by a panel of 16 distinct humans who have knowledge of English poetry and AI writings\. Two slots were assigned as independent and unpaired consisting of 40 unique evaluations total and the remaining 7 slots consisting 140 unique poems were evaluated pairwise using a cross\-dyad peer condition\. The evaluation matrix consisted of 240 instances of AI\-generated poems and 80 human\-authored poetry\. At the macro level, paired human evaluators converged on identical classification for only 35\.71% of the mutually reviewed corpus\.

At the criterion level, agreement was even lower, as shown in Figure[9](https://arxiv.org/html/2607.26221#S4.F9): literary devices reached 27\.14%, punctuation 23\.57%, clarity 22\.86%, originality 22\.14%, and grammar 20\.71%\. This indicates high perceptual subjectivity in human judgment\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/uploads/criterion_wise_agreement.png)Figure 9:Inter\-rater exact agreement rates by survey evaluation metric\.Our observational findings show that generative AI has attained absolute stylistic balance with human poets, revealing severe blindspots in human evaluation\. This is characterized by a strong correlation \(r=0\.68r=0\.68\) between poem clarity and observed originality in Figure[10](https://arxiv.org/html/2607.26221#S4.F10), which severely contributed to an 83\.33% human misclassification rate for human\-authored poems\. In addition, 30 AI\-generated poems attained absolute camouflage with a 0\.00% detection rate, highlighting the complete breakdown of traditional creative limits between human and AI\.

![Refer to caption](https://arxiv.org/html/2607.26221v1/prism-uploads/Code_Generated_Image.png)Figure 10:Linguistic metric correlation heatmap\.TABLE VIII:Human Classification Performance ReportText Origin ClassPrecisionRecallF1\-ScoreSupportAI\-Generated0\.70320\.45420\.5519240Human\-Authored0\.20610\.42500\.277680Global Accuracy0\.4469320Macro Average0\.45460\.43960\.4147320Weighted Average0\.57890\.44690\.4833320

For the 240 true AI evaluations, human evaluators achieved a true positive classification rate of 45\.43% \(110 items\), while misidentifying 54\.16% \(130 items\) as human poems\. For the 80 human\-authored poems, evaluators correctly identified poem origin in only 42\.50% of cases \(34 items\), while misidentifying 57\.50% \(46 items\) as AI\-generated text\.

As shown in Table[VIII](https://arxiv.org/html/2607.26221#S4.T8), the overall human classification accuracy was 44\.69%\.The imprecision profile \(AI: 0\.7032; Human: 0\.2061\) is primarily a result of imbalanced class support in the experimental corpus\. The important metric for assessing human capability is recall, which is balanced yet suppressed across both AI recall at 45\.42% and human recall at 42\.50%\. The severely low F1\-score for human detection \(0\.2776\) establishes that generative language models have attained stylistic consistency with human authors\.

### IV\-DResults of Gemma Classification

In zero\-shot classification, Gemma 4 achieved stronger classification performance than the traditional detectors outpermoning the next best model \(Log\-Likelihood\) by 3\.5% inWeighted F1score\.

TABLE IX:Gemma 4 Classification PerformanceEvaluation DomainPrecisionRecallF1\-ScoreSupportAI Generated Poems0\.9330\.9330\.9334457Human Original Poems0\.8020\.8020\.8021513Overall Accuracy0\.9005970Macro Average0\.8680\.8680\.8685970Weighted Average0\.9000\.9000\.9005970

![Refer to caption](https://arxiv.org/html/2607.26221v1/images/gemma_evasion_ranking.png)Figure 11:Number of False Negative classification produced by LLMs\.Qwen evaded Gemma the most \(Fig\.[11](https://arxiv.org/html/2607.26221#S4.F11)\), which is consistent with the word diversity results\. GPT\-OSS\-120B and Llama\-3\.3\-70B were classified more efficiently, while Qwen’s concrete and distinctive phrasing made it more difficult to detect\.

TABLE X:Modal Criterion Scores for Correct Classifications, Positive \(AI\)TABLE XI:Modal Criterion Scores for Prediction Errors, Positive \(AI\)At the Table[X](https://arxiv.org/html/2607.26221#S4.T10)for true positive classification, modal frequency indicated for the Criteria, Literary Devices and Originality Gemma frequently judged a poem to be AI and for the Criterion Punctuation and Spacing Gemma frequently judged a poem to be Human\. Therefore,

- •For TP predictions,Literary DevicesandOriginalitywas most helpful andPunctuation and Spacingwas the least helpful\.
- •For true negative classification,Literary DevicesandPunctuation and Spacingwere the most helpful\.
- •Moreover, the Table[XI](https://arxiv.org/html/2607.26221#S4.T11)indicates for false positive classifications,Literary DevicesandOriginalityhindered Gemma andPunctuation and Spacinghelped Gemma\.
- •Finally, for false negative classifications,Punctuation and Spacingonly hindered Gemma\.

\.

Hence, in both hard positive and hard negative classifications, the criterionLiterary Deviceshad the most impact\.

### IV\-EResults of the Traditional Detectors

Among the traditional detectors \(See Table[XII](https://arxiv.org/html/2607.26221#S4.T12)\), Log\-Likelihood\[[12](https://arxiv.org/html/2607.26221#bib.bib12)\]performed best, but it was still less robust than Gemma 4 and Log Rank Ratio \(LRR\)\[[11](https://arxiv.org/html/2607.26221#bib.bib11)\], Binoculars\[[10](https://arxiv.org/html/2607.26221#bib.bib10)\]performed second best and the third best respectively\. Finally, Fast\-DetectGPT\[[9](https://arxiv.org/html/2607.26221#bib.bib9)\]performed the worst, contradicting the expectation that DetectGPT\-derived approaches would be resilient against non\-paraphrased AI\-generated poetry\.

TABLE XII:Detailed Classification Reports for AI Text Detectors
### IV\-FQualitative Attributes of GenAI and Human\-Authored Poems

Correctly classified GenAI poems often showed rigid rhyme schemes, symmetrical metaphors, generic sentimental phrasing, and predictable literary devices\. However, some GenAI poems were misleading because they used non\-cliché metaphors, colloquial contractions, irregular grammar, and human\-like stylistic imperfections\.

Human\-authored poems were often identifiable through non\-standard grammar, text\-speak abbreviations, idiosyncratic metaphors, unconventional phrasing, and non\-linear emotional arcs\. However, some human poems were misclassified because they used cliché metaphors, predictable rhyme schemes, polished structures, and AI\-like vocabulary\. These findings show that both human\-like GenAI attributes and AI\-like human attributes contribute to classification errors\. Detailed elaboration is presented at Table[XIII](https://arxiv.org/html/2607.26221#S4.T13)\.

TABLE XIII:Percentage of Gemma’s classification reasoning factors for different scenarios

## VConclusion

The Conclusion is divided into 3 subsections, A\. Findings B\. Limitations and C\. Future Work\. The further elaboration of the aforementioned subsections are stated as,

### V\-AFindings

The key findings of this research remain,

- •Classification Performance: Gemma 4 performed best by the holding bestWeighted F1 Average, exceeding in every classification criteria, albiet by a small margin\. This performance suggests that Gemma 4\-31B can be light weight and robust alternative to the current for\-profit detection tools\. Additionally, Fast\-DetectGPT’s poor performance contradicts the findings of our literature review\. As Fast\-DetectGPT is derived from DetectGPT\[[1](https://arxiv.org/html/2607.26221#bib.bib1)\]hence, it was supposed to robust against our non\-paraphrased Poem dataset but amongst all the AI based detectors, it acheieved the lowestWeighted F1 Averagescore of 0\.552\. Finally, human classification was also poor with 0\.70 precision and 0\.45 recall in AI generated portion and only 0\.20 precision and 0\.42 recall score in human\-written part, affirming the indistinguishability AI poems to human poems\[[3](https://arxiv.org/html/2607.26221#bib.bib3)\]\.
- •Human\-like and Non\-Human\-like Attributes: In our reserach we were able to find the Human\-like and Non\-Human\-like attributes \(Defining Attributes\) and contrastingly we were also able to find Misleading attributes\. In most cases, the rigid and the cliché nature of GenAI compostion aided significantly in detection\. Seemingly, the opppsite is also true\. GenAI composition with unique usage of literary devices \(Mainly, Metaphors and Imageries\) aided in evasion\. Qwen leveraged this misleading attribute to evade Gemma 4 the most\. Similarly, usage of unconventional metaphors, unique grammartical structure and spelling were also part of prominent misleading attributes \(See Example:[A\-A](https://arxiv.org/html/2607.26221#A1.SS1)\)\. Finally, human writing in predictable rhyme and generic patterns is also in the risk of misclassification \(See Example:[A\-B](https://arxiv.org/html/2607.26221#A1.SS2)\)\. Hence, the misclassification via misleading attributes needs mending\.
- •Biased towards Few\-shot examples over User Prompt: In generating, the 3 LLMs had the tendency to be biased towards few\-shot examples over user prompts\. This assumption is supported by the semantic consistency analysis\. In the generation loop, themes were kept independent to what the authors writing style to ensure better lexical diversity and originality but the disconnection between the user given poetry theme and the generated poem’s theme made such separation dificult achieve\. Similar phenomena also occured when in Style Guide, it was explicitly mentioned not to use ”em” dashes \(Subsection[III\-A2](https://arxiv.org/html/2607.26221#S3.SS1.SSS2.Px1)\) but human authors’ usage of ”em” dashes in few\-shot example overruled the style based prompt command\. Therefore, we had to extensively clean the ”em” dashes in the Data Pre\-processing section\.
- •Lack of Word Diversity: Biased towards fewshot blocks can be a reason behind such excessive word repetition as the few\-shots examples were picked randomly for each generation based on the author\. Hence, it is possible that same few\-shots examples were picked excessive amount of times by Python’s random library and as the models exhibit more biased towards few\-shots examples over prompt based themes so such lack of theme adherence decreased word diversity and forced repetitive generations\.

### V\-BLimitations

The limitations of this study include inadequate cleaning of the dataset\. Even after pre\-processing, the potential ambiguity in the constructed dataset cannot be fully ruled out\. Additionally\. during the generation of the final dataset, certain uncertainties and biasness in the universal style guide were noticed\. For example, the style guide instructed not to add any em dashes along with other restrictions\. However, analysis of the AI\-generated poetries showed that some models still generated poems containing em dashes which was later removed through preprocessing\. Moreover, while conducting the survey, another input called ’Overall Label’ should have been taken along with other five evaluation criteria\. Additionally, as our research is based on English poetries, the response from native English speakers should have been taken as well during the survey\. Furthermore, humanizing GenAI texts is crucial to evade the detection tools\. Yet, we did not use any paraphrasers or humanizers in our dataset as implementing humanizing framework in poetry is extremely difficult without damaging the poem integrity by altering the structure and the context of the poem\.

### V\-CFuture Work

Future work remains to train Gemma 4\-31B with hard negative examples \(i\.e\., poems with misleading attribute\)\. Training the model only on such attributes will not only reduce the training margin needed but also gives us a clearer picture of which framework is truly better while detecting GenAI poetry\. Finally, Qwen 3\-32B’s high evasiveness needs further investigation via adding new detectors in the classification pipeline\.

## Appendix ASupplementary Dataset Details

### A\-AUsage of non\-cliché metaphors and imagery by Qwen 3\-32B

Silent Room

A coat hangs on the back of the chair where she once sat, knitting silence into the air\. The clock stopped when she left\. Dust motes drift like forgotten seconds\. I open her jar of perfume a breath of bergamot, then nothing\. The teacup cools beside the book she never finished\. Outside, the wind wears her laugh like a borrowed coat\. I close the door and let the dark collect itself into a shape that fits the empty\.

### A\-BPredictable rhyme scheme and generic writing by a human author

I really miss what we had, not how it all fell apart, Not the final goodbye or the ache in my heart,

I do not wanna touch the ending, I do not want to rewrite the pain, I honor the love that still leaves a soft stain, We were real in those moments, the laughter, the light, The way being seen felt so warm, so right, No damage, no anger, no need to pretend, Just proof that our love did not break, it just bent, We found each other once, and that still feels true, Like a mark on my soul shaped a little like you, Some loves are not forever, but that does not mean, They were not something holy, or deeply unseen, So I carry the knowing, quiet and kind, That love once found us still lives in my heart and mind, And maybe in another life, in another way, What we were together gets the chance to stay\.

## Appendix BExample of Final Generated Prompt

Example of the fully constructed prompt for the author,Garrie Grantwith the themeNature:

STYLE GUIDE: 1\. TITLE: Provide a meaningful, original TITLE at the very top of the poem\. 2\. CONSTRAINTS: Only English language poems are allowed\. Do NOT use em dashes \(—\)\. 3\. ORIGINALITY: Do NOT plagiarize or directly copy the provided author’s work\. Absorb their stylistic voice, but generate a 100% original poem based on the given theme\. Here are 2 examples of Garrie Grant’s writing style\. Use these ONLY to understand their voice, rhythm, and formatting\. Do NOT copy the content\. Write a completely new poem based on the assigned theme\. — EXAMPLE 1 — TITLE: Drinking without occasion on a Saturday night in March\. POEM: The sound of her voice like… The street lights reflecting on a perfect… — END EXAMPLE — — EXAMPLE 2 — TITLE: Afterglow POEM: Where did all the…\. This darkness has no… — END EXAMPLE — Generate a brand new, original poem based on the following theme\. THEME: Nature

## Acknowledgment

The authors thank the participating poets and survey respondents for their contributions to the dataset and evaluation process\.

## References

- \[1\]E\. Mitchell, Y\. Lee, A\. Khazatsky, C\. D\. Manning, and C\. Finn, “DetectGPT: Zero\-shot machine\-generated text detection using probability curvature,” inProc\. ICML, 2023\.
- \[2\]L\. Dugan, A\. Zhu, F\. Alam, P\. Nakov, M\. Apidianaki, and C\. Callison\-Burch, “GenAI Content Detection Task 3: Cross\-Domain Machine\-Generated Text Detection Challenge,”arXiv preprint arXiv:2501\.08913, 2025\.
- \[3\]B\. Porter and E\. Machery, “AI\-generated poetry is indistinguishable from human\-written poetry and is rated more favorably,”Scientific Reports, vol\. 14, no\. 1, p\. 26133, 2024\.
- \[4\]Y\. Chen, H\. Gröner, S\. Zarrieß, and S\. Eger, “Evaluating diversity in automatic poetry generation,”arXiv preprint arXiv:2406\.15267, 2024\.
- \[5\]T\. Chakrabarty, X\. Zhang, S\. Muresan, and N\. Peng, “MERMAID: Metaphor generation with symbolism and discriminative decoding,”arXiv preprint arXiv:2103\.06779, 2021\.
- \[6\]OpenAI, “GPT\-OSS\-120B model card,” 2025\.
- \[7\]Meta AI, “Llama 3\.3 70B model card,” 2024\.
- \[8\]Qwen Team, “Qwen3 technical report and model card,” 2025\.
- \[9\]G\. Bao, Y\. Zhao, H\. Teng, L\. Yang, and Y\. Zhang, “Fast\-DetectGPT: Efficient zero\-shot detection of machine\-generated text via conditional probability curvature,” inProc\. ICLR, 2024\.
- \[10\]A\. Hans et al\., “Spotting LLMs with Binoculars: Zero\-shot detection of machine\-generated text,” 2024\.
- \[11\]J\. Su, T\. Zhuo, D\. Wang, and P\. Nakov\. 2023\. ”Detectllm: Leveraging log rank information for zero\-shot detection of machine\-generated text\.” InFindings of the Association for Computational Linguistics: EMNLP 2023, 12395–12412\.
- \[12\]I\. Solaiman et al\., “Release strategies and the social impacts of language models,” 2019\.
- \[13\]Google DeepMind\. 2026\. ”Gemma 4 model card — Google AI for Developers\.”[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)\.
- \[14\]B\. Li, H\. Wang, and H\. Wilkinson\.Poemetric: The last stanza of humanity\.*arXiv preprint arXiv:2604\.03695*, 2026\.
- \[15\]S\. Wang, F\. Ye, H\. Lyu, C\. Gou, J\. Wu, J\. Yao, C\. Xu, J\. Luo, and D\. F\. Wong, “Seeing the poem: Image\-semantic detection of AI\-generated modern Chinese poetry with MLLMs,”arXiv preprint arXiv:2605\.22654, 2026\.