Using Poly-Encoders for Computationally Efficient Automated Creativity Assessment

# Using Poly-Encoders for Computationally Efficient Automated Creativity Assessment
Source: [https://arxiv.org/html/2608.26165](https://arxiv.org/html/2608.26165)
11institutetext:Wheeler Magnet High School, Marietta, Georgia, USA
11email:sam\.grouchnikov@gmail\.com22institutetext:Kennesaw State University, Marietta, Georgia, USA
22email:pgrego10@students\.kennesaw\.edu, jnoh3@kennesaw\.edu###### Abstract

Automated creativity assessment has been a long standing challenge, with traditional methods often being resource intensive or lacking practical accuracy\. We introduce a novel approach by using Poly\-Encoder for computationally efficient and accurate automated creativity assessment\. We fine\-tuned a Poly\-Encoder on a public dataset from the Scientific Creative Thinking Test, comprised of approximately 18,000 human\-rated question responses\. Our method leverages small pre\-trained BERT encoders, achieving performance comparable to fine\-tuned Large Language Models while significantly reducing computational demands\. Experiments with the BERT\-family models and poly\-code counts achieved Pearson correlations of up tor=0\.74,95%​CI​\[0\.73,0\.75\]r=0\.74,95\\%\\text\{ CI \}\[0\.73,0\.75\]with human raters, matching the performance of resource intensive LLMs\. This study bridges the gap between high performance and computational efficiency, potentially enabling widespread implementation of automated creativity assessment on accessible consumer\-grade hardware\. With some limitations, our findings suggest that Poly\-Encoders are a promising alternative to LLMs for practical, scalable creativity assessment in various contexts, especially educational\.

## 1Introduction

Creativity is the basis of human progress, enabling the generation of novel, valuable, and feasible ideas across domains such as science, technology, engineering, and mathematics\. Yet, assessing creativity remains a persistent challenge, highlighted best by creativity’s subjective nature\[[17](https://arxiv.org/html/2608.26165#bib.bib17)\]\. Traditional approaches rely on human judges, proving the process to be both time and resource\-intensive\[[10](https://arxiv.org/html/2608.26165#bib.bib10),[1](https://arxiv.org/html/2608.26165#bib.bib1),[12](https://arxiv.org/html/2608.26165#bib.bib12)\]\. Even with shared definitions and common rubrics, inconsistency often emerges between evaluators\[[10](https://arxiv.org/html/2608.26165#bib.bib10),[7](https://arxiv.org/html/2608.26165#bib.bib7),[3](https://arxiv.org/html/2608.26165#bib.bib3)\]\. This raises the central question: how can artificial intelligence be used to automate creativity assessment while still capturing the human creative intent?

Effective AI\-based creativity assessment requires managing test complexity, efficient deployment, and alignment with human intent\[[23](https://arxiv.org/html/2608.26165#bib.bib23)\]\. Transformers are ideal for this task as they are designed to accurately contextualize text, and capture complex semantic nuances\[[15](https://arxiv.org/html/2608.26165#bib.bib15)\]\. Through fine\-tuning, such transformers may be trained to identify linguistic patterns associated with various types of creativity\[[6](https://arxiv.org/html/2608.26165#bib.bib6),[21](https://arxiv.org/html/2608.26165#bib.bib21)\]\.

In scientific contexts, creativity involves adual\-space search: generating hypotheses from memory and solving problems by generalizing from experiments\[[4](https://arxiv.org/html/2608.26165#bib.bib4),[19](https://arxiv.org/html/2608.26165#bib.bib19)\]\. Automated models must therefore understand scientific reasoning while aligning with psychometric criteria including fluency, originality, and elaboration\[[11](https://arxiv.org/html/2608.26165#bib.bib11),[13](https://arxiv.org/html/2608.26165#bib.bib13)\]\. This is particularly important in education, where creativity\-based assessments push beyond strict memorization to test divergent thinking and problem\-solving\[[5](https://arxiv.org/html/2608.26165#bib.bib5),[20](https://arxiv.org/html/2608.26165#bib.bib20)\]\.

However, traditional human evaluation scales poorly\. While Large Language Models \(LLMs\) offer a solution, they can be unreliable and sensitive to small input changes\[[9](https://arxiv.org/html/2608.26165#bib.bib9)\]\. Furthermore, reliance on cloud\-based LLMs like GPT\-4 introduces significant data privacy risks and recurring API costs that may be prohibitive for school districts\. In contrast, local deployment ensures that sensitive student data remains within the school’s secure infrastructure\. Efficient alternatives like Poly\-Encoders\[[16](https://arxiv.org/html/2608.26165#bib.bib16)\]provide a more reliable, scalable, and accessible path for real\-time assessments using consumer\-grade hardware available in classrooms\.

In this study, the implementation and analysis of the Poly\-Encoder\[[16](https://arxiv.org/html/2608.26165#bib.bib16)\]is used to answer the following questions:

1. 1\.How can a Poly\-Encoder be trained to assess creativity instead of similarity?
2. 2\.How computationally efficient are Poly\-Encoders, and can they be used on consumer hardware?

## 2Related Work

### 2\.1Evolution of Creativity Assessment

Early automated assessments relied on semantic distance models such as SemDis and GloVe\. While effective at measuring semantic \(dis\)similarity through word\-level relationships, they fail to capture semantic nuance and sentence\-level contexts, both of which are essential for creativity assessment\[[21](https://arxiv.org/html/2608.26165#bib.bib21)\]\. This resulted in a shift towards the utilization of Large Language Models \(LLMs\) that embed meaning at the sentence level and beyond\. Initial prompt\-based methods using GPT\-4 achieved Pearson correlations between 0\.2 and 0\.67 with human ratings for novelty and feasibility\[[18](https://arxiv.org/html/2608.26165#bib.bib18)\]\. However, these systems struggle with absolute scoring, and lack reliability due to their reliance on calling external APIs\[[22](https://arxiv.org/html/2608.26165#bib.bib22)\]\.

### 2\.2Fine\-Tuning Pre\-Trained Language Models

The next wave of methods used fine\-tuning, training language models on questions with responses scored by human raters\. Fine\-tuning GPT\-3 and T5 on 27,000 alternative uses task responses yielded correlations of up tor=0\.81r=0\.81\[[21](https://arxiv.org/html/2608.26165#bib.bib21)\]\. Similarly, the Scientific Creative Thinking Test \(SCTT\) utilized a dataset of 18,000 responses to fine\-tune LLaMA\-2\-7b, reaching a Pearson correlation ofr=0\.74r=0\.74\[[4](https://arxiv.org/html/2608.26165#bib.bib4)\]\. Although these methods exhibited strong scoring capability, their dependence on LLMs and the requirement for re\-encoding questions/responses makes them computationally sub\-optimal

### 2\.3The Poly\-Encoder Alternative

Poly\-Encoders\[[16](https://arxiv.org/html/2608.26165#bib.bib16)\]provide a middle ground between the efficiency and semantic distance and the accuracy of fine\-tuned LLMs\. Poly\-Encoders project context \(or query\) embeddings into multiple spaces by using a fixed\-size set of learnable global attention codes\[[2](https://arxiv.org/html/2608.26165#bib.bib2),[24](https://arxiv.org/html/2608.26165#bib.bib24)\], providing potential to learn deep semantic representations of text and different facets of creativity without the computational overhead of LLMs\. When paired with lightweight encoders like BERT\[[8](https://arxiv.org/html/2608.26165#bib.bib8)\], poly\-encoders deliver consistent, reliable scoring that can be deployed on consumer\-grade hardware: a task impossible with LLMs\.

## 3Methods

Due to the potential applications and implications of accurate automated creativity assessment within educational and professional environments, a Poly\-Encoder was adopted and trained to evaluate responses to questions\. These questions were specifically centered around scientific prompts, including research inquiries and hypothesis formulation\. The Poly\-Encoder was implemented utilizing PyTorch Lighting, and training was executed in a distributed data parallel \(DDP\) configuration across three Nvidia RTX 3090 GPUs\.

### 3\.1Datasets and Training Schematics

All training and testing data were obtained from the public repository released by the authors of the Scientific Creativity Thinking Task \(SCTT\) validation study111[https://osf\.io/preprints/psyarxiv/y5fbs\_v1](https://osf.io/preprints/psyarxiv/y5fbs_v1)\. The dataset contained approximately 18,000 responses, each paired with human\-assigned ground\-truth creativity labels corresponding to specific prompts\. There are 15 prompts, each with approximately 1,200 responses\. In our system, the prompt is the input to the Poly\-Encoder as the context, while the response is the input as the candidate \(see Figure[1](https://arxiv.org/html/2608.26165#S3.F1)\)\. To confirm the validity and reliability of the SCTT test, the authors employed multiple statistical tests\. Inter\-rater reliability achieved a coefficient ofw=0\.85w=0\.85and a test\-retest analysis yielded a moderate temporal stability with a Pearson correlation ofr=0\.67r=0\.67across a one month interval\. A conventional 70/10/20 train/validation/test split was employed \(at the response level\)\. Prompts in the training set were also included in the test set, while responses in the training set were not used in the test set\. Within prompts, 70% of responses were allocated to the training set, 10% to validation, and 20% to testing\.

#### 3\.1\.1BERT for Context/Candidate Embeddings

Bidirectional Encoder Representations from Transformers \(BERT\) serves as a robust foundation for semantic representation learning\[[8](https://arxiv.org/html/2608.26165#bib.bib8)\], giving a favorable combination of learning potential and computational efficiency\. To capture nuanced representations of creativity, multiple BERT\-family models were explored for context and candidate encodings, gradually scaling in size to investigate performance trade\-offs \(Table[1](https://arxiv.org/html/2608.26165#S3.T1)\)\.

Table 1:Tested BERT Models with their Sizes and Parameter Counts
#### 3\.1\.2Testing Poly\-Code Counts

As described in the original Poly\-Encoder paper, varying the number of poly\-codes \(mm\) can influence performance and computational cost\. Ideally, reducingmmshould not substantially degrade results, indicating efficiency in representation\. All hyper\-parameters aside from poly\-code count were held constant during testing\. Tested poly\-code counts were 64, 128, 256, and 512\.

### 3\.2Architectural Tweaks

The base architecture of the Poly\-Encoder was adopted from the paper released by Humeau et al\.\[[16](https://arxiv.org/html/2608.26165#bib.bib16)\]\. The flow of information and calculations are depicted in Figure[1\(a\)](https://arxiv.org/html/2608.26165#S3.F1.sf1)The original Poly\-Encoder architecture produces a similarity score by computing the dot product between the poly\-context and the candidate embeddings\. While effective for retrieval and matching tasks, this mechanism underperformed for deeper scoring tasks, such as in our case, creativity\. To address this, the final score was obtained by passing the poly\-context embeddings through a lightweight regression head, allowing the model to recognize non\-linearities of the final embeddings\. This head also processed candidate vectors and the element\-wise products of poly\-context and candidate vectors, addressing the model’s initial difficulty in differentiating between candidates\. The regression head itself consisted of a multiple linear layers, along with an activation function and dropout\. This architectural adjustment is visually illustrated in Figure[1\(b\)](https://arxiv.org/html/2608.26165#S3.F1.sf2)\.

![Refer to caption](https://arxiv.org/html/2608.26165v1/pe-arch.png)\(a\)Original Poly\-Encoder\.
![Refer to caption](https://arxiv.org/html/2608.26165v1/context-sequence.png)\(b\)Modified Poly\-Encoder\.

Figure 1:Comparison of architectures \(original vs our modified version\)

## 4Results

### 4\.1BERT Comparisons

Progressively larger BERT models were tested while keeping all other aspects of the model constant\. To ensure correct convergence, a plot was obtained from the Weights & Biases logger\. These plots are shown in Figure[2\(a\)](https://arxiv.org/html/2608.26165#S4.F2.sf1)\. The final highest correlations achieved by the BERT models are shown in Table[2](https://arxiv.org/html/2608.26165#S4.T2)\. DeBERTa\-v3\-Large achieved the strongest correlation with human creativity scores\(p<0\.001\)\(p<0\.001\), outperforming all other tested encoders\. This result aligns with expectations given DeBERTa’s disentangled attention mechanism\[[14](https://arxiv.org/html/2608.26165#bib.bib14)\], which models content and positional information through separate vectors\. However, RoBERTa variants demonstrated comparable results, suggesting that smaller, computationally efficient models may achieve similar accuracy\. It is important to note however, all of the poly\-code counts producedp<0\.05p<0\.05between DeBERTa and RoBERTa variants\.

### 4\.2Poly\-Code Count Comparisons

Using DeBERTa\-v3\-Large as the encoder, poly\-code counts of 64, 128, 256, and 512 were evaluated, with test correlation plots shown in Figure[2\(b\)](https://arxiv.org/html/2608.26165#S4.F2.sf2)\. Additionally, RoBERTa\-Base was chosen for poly\-code count comparisons due to its desirable mix of low parameter count and strong correlation\. Comparisons are shown in Table[2](https://arxiv.org/html/2608.26165#S4.T2)\. It should be noted, DeBERTa\-v3\-Large in a Poly\-Encoder achieved the same correlation as LLaMA\-2\-7B\[[4](https://arxiv.org/html/2608.26165#bib.bib4)\]while using a fraction of the resources as LLaMA\.

![Refer to caption](https://arxiv.org/html/2608.26165v1/bert-test-corr-nt.png)\(a\)BERT Model Comparison
![Refer to caption](https://arxiv.org/html/2608.26165v1/deberta-polym-test-corr.png)\(b\)Poly\-code \(mm\) Comparison

Figure 2:Training convergence and Pearson correlation resultsTable 2:Performance comparison across poly\-code counts \(mm\) with 95% CIThe model’s predicted scores versus human ground\-truth ratings are shown in Figure[3\(a\)](https://arxiv.org/html/2608.26165#S4.F3.sf1)\. The model demonstrated strong alignment with human ratings, correctly predicting the majority of creativity scores within reasonable threshhold \(% difference between ground\-truth and predicted scores\), as shown in Table[3\(b\)](https://arxiv.org/html/2608.26165#S4.F3.sf2)\. Additionally, with DeBERTa\-v3\-Large, the mean absolute error \(MAE\) between ground\-truth and predicted scores was 0\.0077, further proving the accuracy of the model\.

![Refer to caption](https://arxiv.org/html/2608.26165v1/final-scatterplot-nt.png)\(a\)Human vs Model Scores
\(b\)Model Scoring Threshold Accuracy

Figure 3:Performance Analysis: \(a\) visualizes the correlation between predicted and ground\-truth scores, while \(b\) provides the exact cumulative accuracy at various error thresholds\.
### 4\.3Efficiency, Scoring Times

Although not performing better in terms of correlation from previous approaches, the true benefit of Poly\-Encoders comes in their efficiency\. The final scoring times \(per candidate\) for the DeBERTa\-v3\-Large and RoBERTa\-Model were 0\.010 and 0\.0022 seconds on a CPU \(Intel i5, common consumer\-grade CPU\) respectively\.

## 5Conclusions

Performance and Efficiency: While Poly\-Encoders do not outperform large language models in terms of absolute correlation, they achieve comparable performance with significantly higher computational efficiency\. By encoding prompts once and utilizing lightweight BERT\-based encoders, Poly\-Encoders bypass the massive memory requirements presented by large models such as GPT\-3 and LLaMA\-2\. This architecture allows for sub\-second scoring on consumer grade hardware: a feature necessary for real\-world deployment and use in classrooms with limited resources\.

Poly\-Code Impact: Consistent with existing literature, variations in the number of poly\-codes \(mm\) had no significant effect on model performance\. This confirms that poly\-codes enhance representational power while having a negligible effect on model size\. The only significant affecting factor of performance was the family and size of the BERT encoder used \(p<0\.05p<0\.05between BERT families with same poly\-code counts\)\.

Implications and future work: Poly\-Encoders lay a framework for practical implementation in classrooms due to their accessibility on standard hardware and local\-run privacy benefits\. Future research should explore multidimensional assessment \(novelty, feasibility, value\) and generalize to other tasks such as the Alternative Uses Task \(AUT\)\. Additionally, incorporating textual justifications for model scores could further enhance accuracy and interpretability\. To support reproducibility, all code and data are available online222https://github\.com/sam\-grouchnikov/ca\-polyencoder\-official\.

Limitations: This study was limited by the SCTT’s dataset’s label noise and the model’s inability to generalize to unseen prompts\. Potential biases include models preferring longer responses, and responses with writing styles from less common dialogues\. BERT is pretrained on “Standard English” and thus lower scores may be assigned to responses that deviate from this\. Such language may include African American Vernacular English \(AAVE\), regional dialects, or non\-standard syntax\. While fine\-tuning BERT is more efficient than training LLMs, it still requires basic GPU access, which may limit some users\. Most importantly, AI\-based creativity assessment methods continue to function as “black boxes\.” It remains difficult to fully interpret how and why such models assign specific creativity scores\. Enhancing model interpretability represents a crucial area for future investigation, as results without transparent reasoning provide an incomplete understanding of creativity evaluation\.

## References

- \[1\]Acar, S\., Organisciak, P\., Dumas, D\.: Automated Scoring of Figural Tests of Creativity with Computer Vision\. The Journal of Creative Behavior \(2024\)
- \[2\]Bahdanau, D\., Cho, K\., Bengio, Y\.: Neural Machine Translation by Jointly Learning to Align and Translate\. CoRRabs/1409\.0473\(2014\)
- \[3\]Barth, P\., Stadtmann, G\.: Creativity assessment over time: Examining the reliability of CAT ratings\. The journal of creative behavior55\(2\), 396–409 \(1 Jun 2021\)\. https://doi\.org/10\.1002/jocb\.462
- \[4\]Beaty, R\., Cortes, R\.A\., Luchini, S\., Patterson, J\.D\., Forthmann, B\., Baker, B\.S\., Barbot, B\., Hardiman, M\., Green, A\.: The scientific creative thinking test \(SCTT\): Reliability, validity, and automated scoring\. PsyArxiv Preprints \(2024\)
- \[5\]Beghetto, R\.A\.: Creativity in the classroom\. In: Kaufman, J\.C\., \(Eds\. \)\., S\.R\. \(eds\.\) The Cambridge handbook of creativity\. Cambridge University Press \(2010\)
- \[6\]Bellemare\-Pepin, A\., Lespinasse, F\., Thölke, P\., Harel, Y\., Mathewson, K\., Olson, J\.A\., Bengio, Y\., Jerbi, K\.: Divergent creativity in humans and Large Language Models\. arXiv \[cs\.CL\] \(1 Jul 2025\)
- \[7\]Cropley, D\.H\., Kaufman, J\.C\.: Measuring functional creativity: Non\-expert raters and the creative solution diagnosis scale\. The journal of creative behavior46\(2\), 119–137 \(1 Jun 2012\)\. https://doi\.org/10\.1002/jocb\.9
- \[8\]Devlin, J\., Chang, M\.W\., Lee, K\., Toutanova, K\.: BERT: Pre\-training of deep bidirectional Transformers for language understanding\. arXiv \[cs\.CL\] \(10 Oct 2018\)\. https://doi\.org/10\.48550/arXiv\.1810\.04805
- \[9\]Dhinakaran, A\.: Why you should not use numeric evals for LLM as a judge\.[https://arize\.com/blog\-course/numeric\-evals\-for\-llm\-as\-a\-judge/](https://arize.com/blog-course/numeric-evals-for-llm-as-a-judge/), accessed: 2025\-10\-22
- \[10\]Dumas, D\.G\., Organisciak, P\., Doherty, M\.: Measuring divergent thinking originality with human raters and text\-mining models: A psychometric comparison of methods\. Psychology of Aesthetics, Creativity, and the Arts \(2020\)
- \[11\]Glover, J\., Gary, A\.L\.: Procedures to increase some aspects of creativity\. Journal of applied behavior analysis9\(1\), 79–84 \(1 Mar 1976\)\. https://doi\.org/10\.1901/jaba\.1976\.9\-79
- \[12\]Haase, J\., Hanel, P\.H\.P\., Pokutta, S\.: S\-DAT: A multilingual, GenAI\-driven framework for automated divergent thinking assessment\. arXiv \[cs\.CL\] \(23 Oct 2025\)\. https://doi\.org/10\.48550/arXiv\.2505\.09068
- \[13\]Handayani, S\.A\., Rahayu, Y\.S\., Agustini, R\.: Students’ creative thinking skills in biology learning: fluency, flexibility, originality, and elaboration\. Journal of physics\. Conference series1747\(1\), 012040 \(1 Feb 2021\)\. https://doi\.org/10\.1088/1742\-6596/1747/1/012040
- \[14\]He, P\., Liu, X\., Gao, J\., Chen, W\.: DeBERTa: Decoding\-enhanced BERT with disentangled attention\. arXiv \[cs\.CL\] \(5 Jun 2020\)\. https://doi\.org/10\.48550/arXiv\.2006\.03654
- \[15\]Hill, F\.: Why transformers are obviously good models of language\. arXiv \[cs\.CL\] \(7 Aug 2024\)
- \[16\]Humeau, S\., Shuster, K\., Lachaux, M\.A\., Weston, J\.: Poly\-encoders: Transformer architectures and pre\-training strategies for fast and accurate multi\-sentence scoring\. arXiv \[cs\.CL\] \(21 Apr 2019\)
- \[17\]Katz, A\., Giacommelli, L\.: The subjective nature of creativity judgments\. Bulletin of the psychonomic society \(1982\)\. https://doi\.org/10\.3758/BF03334789
- \[18\]Kern, F\.B\., Wu, C\.T\., Chao, Z\.C\.: Assessing novelty, feasibility and value of creative ideas with an unsupervised approach using GPT\-4\. British journal of psychology \(London, England: 1953\) \(22 Jul 2024\)\. https://doi\.org/10\.1111/bjop\.12720
- \[19\]Klahr, D\., Dunbar, K\.: Dual space search during scientific reasoning\. Cognitive science12\(1\), 1–48 \(1 Jan 1988\)\. https://doi\.org/10\.1207/s15516709cog1201\_1
- \[20\]Lucas, B\., Claxton, G\., Spencer, E\.: Progression in student creativity in school: First steps towards new forms of formative assessments \(2013\)
- \[21\]Organisciak, P\., Acar, S\., Dumas, D\., Berthiaume, K\.: Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models\. Thinking skills and creativity49\(101356\), 101356 \(Sep 2023\)\. https://doi\.org/10\.1016/j\.tsc\.2023\.101356
- \[22\]Rabeyah, A\.A\., G’oes, F\., Volpe, M\., Medeiros, T\.: Do LLMs Agree on the Creativity Evaluation of Alternative Uses? ArXivabs/2411\.15560\(2024\)
- \[23\]Rafner, J\., Beaty, R\.E\., Kaufman, J\.C\., Lubart, T\., Sherson, J\.: Creativity in the age of generative AI\. Nature human behaviour7\(11\), 1836–1838 \(Nov 2023\)\. https://doi\.org/10\.1038/s41562\-023\-01751\-1
- \[24\]Vaswani, A\., Shazeer, N\., Parmar, N\., others: Attention is all you need