Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

# Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
Source: [https://arxiv.org/html/2609.03432](https://arxiv.org/html/2609.03432)
Jin WuAffiliation:Shanghai Institute of Artificial Intelligence for Education, East China Normal UniversityAffiliation:School of Computer Science and Technology, East China Normal UniversityXiaoyu LiAffiliation:School of Education and Intelligent Education Research Center, Yangzhou UniversityChanjin ZhengAffiliation:Shanghai Institute of Artificial Intelligence for Education, East China Normal UniversityYifeng ZhouAffiliation:School of Data Science and Engineering, East China Normal University\{51274118009, 52275901018\}@stu\.ecnu\.edu\.cn,chjzheng@dep\.ecnu\.edu\.cn

###### Abstract

Automated evaluation of creativity tasks remains challenging for LLM\-as\-a\-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias\. Such limitations are particularly evident in Contextually\-Grounded and Procedurally\-Structured Tasks \(CGPST\), a complex multi\-step creativity task where inter\-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments\. Existing approaches either rely on task\-specific training or directly apply LLM\-as\-a\-Judge, both of which struggle to ensure reliable evaluation under such complexity\. To bridge these gaps, we proposeCreaEval, an automatedcreativityevaluator for CGPST that decouples typical LLM\-as\-a\-Judge into analysis and judging\. Correspondingly, CreaEval involves two critical phases:Memory\-augmented Analysis, a SoT\-LLM converts multi\-step responses into structured evaluation evidence, incorporating cross\-step memory; andEvidence\-based Judging, a Judge\-LLM uses the extracted evidence for judging without accessing raw responses\. Comprehensive experiments show that CreaEval achieves an average performance improvement of 22\.74% over the second\-best baselines across CGPST and two classic simple creativity tasks, demonstrating its generalizability\. The code is available at[https://github\.com/Jaong/CreaEval](https://github.com/Jaong/CreaEval)\.

00footnotetext:Equal contribution\.00footnotetext:Corresponding author\.## 1Introduction

The creativity of Large Language Models \(LLMs\) has attracted increasing attention, with applications in creative writing, novel mathematical reasoning, and other creative domains\([Kumar et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib5);[Lin et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib9);[Ye et al\., 2025b](https://arxiv.org/html/2609.03432#bib.bib10)\)\. However, current automated evaluation methods mainly focus on simple creativity tasks, such as the Alternate Uses Task \(AUT\)\([Lu et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib7)\)and the Torrance Test of Creative Thinking \(TTCT\)\([Kumar et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib5)\), while the automated evaluation of complex multi\-step creativity tasks remains underexplored\.

Among creativity tasks, Contextually\-Grounded and Procedurally\-Structured tasks \(CGPST\)\([Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\)is particularly challenging due to its high complexity, including multi\-step and scenario\-based features\. Each task on CGPST is grounded in a complete future scenario and needs to solve multiple interdependent steps\([Treffinger, 1995](https://arxiv.org/html/2609.03432#bib.bib19);[Treffinger et al\., 2012a](https://arxiv.org/html/2609.03432#bib.bib18)\)\.

Existing automated evaluation methods for the above creativity tasks mainly fall into two categories\. The first involves training task\-specific models, which require additional training resources and annotated datasets, making them costly to develop\([Do et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib20);[Wang and Liu, 2025](https://arxiv.org/html/2609.03432#bib.bib21);[Li and Pan, 2025](https://arxiv.org/html/2609.03432#bib.bib22)\)\. The other is LLM\-as\-a\-Judge, a training\-free approach that has become increasingly dominant due to the strong capabilities of LLMs\([Zheng et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib1);[Li et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib2);[Ye et al\., 2025b](https://arxiv.org/html/2609.03432#bib.bib10)\)\. However, directly applying LLM\-as\-a\-Judge to CGPST yields low agreement, as LLMs are often susceptible to biases such as verbosity bias and leniency bias, making it difficult to accurately capture subtle score differences in subjective dimensions\([Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\)\.

Table 1:Step\-wise Information on CGPST\. See Appendix[A](https://arxiv.org/html/2609.03432#A1)for a full description\.Applying automated evaluation to CGPST presents several key challenges compared to other simple creativity tasks\.\(1\)The task ishighly complexcompared to traditional creativity tasks\. Each task requires evaluating multiple interdependent steps based on a scenarios, with each step assessed across multiple dimensions\.\(2\)The task isinherently subjective, as there are no fixed answers and earlier responses influence subsequent ones, resulting in highly diverse outputs\([Zhao et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib6);[Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\)\.\(3\)Some dimensions involvelarge scoring ranges\(e\.g\., 10\-level scoring\), which further increases scoring instability\.

To address these challenges, we proposeCreaEval, an automatedcreativityevaluator using LLMs in complex multi\-step creativity tasks such as CGPST\. Inspired by human evaluation practices, where raters first analyze responses across each dimension before assigning rubric\-based scores\([Klein et al\., 1998](https://arxiv.org/html/2609.03432#bib.bib26);[Harsch and Martin, 2013](https://arxiv.org/html/2609.03432#bib.bib25)\), CreaEval decouples the evaluation process into two phases:Memory\-augmented AnalysisandEvidence\-based Judging\. In the first phase, a SoT\-LLM incrementally organizes raw responses into structured intermediate evaluation evidence in the form of Structure\-of\-Thought \(SoT\)\([Qi et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib17);[Wang et al\., 2026a](https://arxiv.org/html/2609.03432#bib.bib16)\)\. Considering the interdependencies among CGPST steps, CreaEval further introduces a memory mechanism to maintain cross\-step coherence during analysis\. In the second phase, each Judge\-LLM assigns scores based on the extracted evidence and predefined rubrics, without accessing the original responses, thereby enabling evidence\-grounded scoring\.

Compared with typical LLM\-as\-a\-Judge, this decoupled design constrains judging with structured evidence, thereby narrowing the range of plausible judgments\. This not only improves scoring accuracy and stability, but also mitigates verbosity and leniency biases\. The contributions are threefold:

- •We proposeCreaEval, an novel automatedcreativityevaluator for complex and subjective creativity tasks such as CGPST that decouples typical LLM\-as\-a\-Judge into Memory\-augmented Analysis and Evidence\-based Judging\.
- •Extensive experiments show that CreaEval achieves an average human\-LLM agreement of 0\.64 \(quadratic weighted kappa, QWK\) on CGPST, outperforming the second\-best baseline \(supervised training\) by 0\.17\.
- •Further analysis reveals that the decoupled design enhances scoring stability across dimensions and mitigates verbosity and leniency biases in LLM\-as\-a\-Judge, providing new insights into evaluating creativity tasks\.

## 2Related Work

### 2\.1Creativity Tasks

Traditional creativity tasks originate from educational and psychological studies, such as AUT, which requires participants to generate as many novel uses as possible for a common object\([Lu et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib7);[Zhao et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib6);[Organisciak et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib31);[Hadas and Hershkovitz, 2024](https://arxiv.org/html/2609.03432#bib.bib32)\), and TTCT, which assesses creativity through responses to open\-ended and unconventional scenarios\([Torrance, 1966](https://arxiv.org/html/2609.03432#bib.bib4);[Kumar et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib5)\)\. However, these tasks are typically single\-step and structurally simple, limiting their ability to capture complex creative processes\. Recently,[Wang et al\. \(2026b\)](https://arxiv.org/html/2609.03432#bib.bib11)proposed CGPST, a multi\-step, scenario\-based benchmark for evaluating creative problem\-solving abilities of LLMs\. As shown in Table[1](https://arxiv.org/html/2609.03432#S1.T1), each task is grounded in a complete future scenario and requires LLMs to sequentially solve six interdependent steps, involving diverse abilities\([Treffinger, 1995](https://arxiv.org/html/2609.03432#bib.bib19);[Treffinger et al\., 2012a](https://arxiv.org/html/2609.03432#bib.bib18)\)\. Compared to traditional creativity tasks, CGPST exhibits significantly higher complexity\.

![Refer to caption](https://arxiv.org/html/2609.03432v1/pipeline_2.png)Figure 1:Our proposed CreaEval framework\. CreaEval decouples the evaluation process into two phases:1\) Memory\-augmented Analysis\.SoT\-LLM first extracts intermediate evaluation evidence \(blue\) based on the responses and dimensions in a step\-by\-step manner\.2\) Evidence\-based Judging\.Each Judge\-LLM then performs scoring based on the extracted evidence and predifined rubrics without accessing raw responses\.
### 2\.2LLM\-as\-a\-Judge for Creativity Evaluation

Recently, several studies have explored automated evaluation methods for creativity tasks\([Zheng et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib1);[Liang et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib3);[Li et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib2);[Ye et al\., 2025b](https://arxiv.org/html/2609.03432#bib.bib10)\)\. For example,[Zhao et al\. \(2025\)](https://arxiv.org/html/2609.03432#bib.bib6)leverages GPT\-4 to generate TTCT\-inspired datasets and employs LLMs for scoring responses\.[Lu et al\. \(2024\)](https://arxiv.org/html/2609.03432#bib.bib7)applies LLM\-as\-a\-Judge to AUT tasks, achieving an average human–LLM agreement of 0\.49 \(Kendall’sτ\\tau\) across four dimensions, even surpassing inter\-human agreement \(0\.39\), suggesting the potential of LLM\-as\-a\-Judge to improve scoring reliability in creativity tasks\. However, existing studies mainly focus on relatively simple and traditional creativity benchmarks\. Directly applying LLM\-as\-a\-Judge to more complex tasks such as CGPST remains challenging\. For instance,[Wang et al\. \(2026b\)](https://arxiv.org/html/2609.03432#bib.bib11)shows that a direct LLM\-as\-a\-Judge approach with few\-shot prompting achieves only 0\.31 \(pearson correlation coefficient, PCC\) human–LLM agreement, highlighting the need for more reliable and effective evaluation frameworks for complex multi\-step creativity tasks\.

## 3Methodology

Existing LLM\-as\-a\-Judge fail to decouple analysis from judging, instead directly mapping from raw responsesRs=\{Rts\}t=1NR^\{s\}=\\\{R^\{s\}\_\{t\}\\\}\_\{t=1\}^\{N\}\(NN=6\) to a score listSS\. This leads to the LLM’s inability to effectively capture key intermediate evidence when handling complex creativity tasks\. To address this limitation, we proposeCreaEval, a novel evaluation framework that explicitly models the evaluation process in a structured manner, decoupling analysis and judging\. As illustrated in Figure[1](https://arxiv.org/html/2609.03432#S2.F1), CreaEval consists of two sequential phases: \(1\) Memory\-augmented Analysis; and \(2\) Evidence\-based Judging\.

### 3\.1Phase 1: Memory\-augmented Analysis

Given the raw responseRtsR^\{s\}\_\{t\}and the corresponding dimension descriptionDtD\_\{t\}in Step\-tt, SoT\-LLM performs structured evidence extraction in a step\-by\-step manner\. Specifically, it iteratively processes each step of the CGPST by jointly considering the task scenarioScenario, the step responseRtsR^\{s\}\_\{t\}, and the evaluation dimensionsDtD\_\{t\}\. Within Step\-tt, SoT\-LLM in CreaEval extracts multi\-dimensional evidence represented as a structured mappinget=\{⟨ki,vi⟩\}i=1\|𝒟t\|e\_\{t\}=\\left\\\{\\langle k\_\{i\},v\_\{i\}\\rangle\\right\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\_\{t\}\|\}, where each pair⟨ki,vi⟩\\langle k\_\{i\},v\_\{i\}\\rangledenotes an evaluation dimensionkik\_\{i\}and its corresponding evidenceviv\_\{i\}\. This progressive design prevents SoT\-LLM from being overwhelmed by all subjective information at once, thereby improving the reliability and consistency of the extracted evidence\.

The steps of CGPST exhibit strong temporal dependencies, as responses in later steps are conditioned on decisions made in earlier ones\. For instance, the solution proposed in Step\-3 is designed to address the problem identified in Step\-2\. To model these dependencies, CreaEval introduces a memory mechanism that preserves step\-relevant memory statemtm\_\{t\}\. Specifically, in addition to generating evidenceete\_\{t\}, SoT\-LLM produces a correspondingmt=S​u​m​m​a​r​y​\(Rts,Mt−1\)m\_\{t\}=Summary\(R^\{s\}\_\{t\},M\_\{t\-1\}\), which summarizes the key information fromRtsR^\{s\}\_\{t\}and is passed as contextual input to subsequent steps as shown in Figure[1](https://arxiv.org/html/2609.03432#S2.F1)\. This mechanism retains cross\-step contextual dependencies, thereby augmenting the evidence extraction process and improving accuracy, defined as:

\(et,mt\)=fS​o​T​\(S​c​e​n​a​r​i​o,Rts,Dt,Mt−1\)\(e\_\{t\},m\_\{t\}\)=f\_\{SoT\}\(Scenario,R^\{s\}\_\{t\},D\_\{t\},M\_\{t\-1\}\)\(1\)
whereete\_\{t\}andmtm\_\{t\}denote the evaluation evidence and memory state extracted by SoT\-LLM in Step\-tt, respectively\.D=\{Dt\}t=1ND=\\\{D\_\{t\}\\\}\_\{t=1\}^\{N\}andRs=\{Rts\}t=1NR^\{s\}=\\\{R^\{s\}\_\{t\}\\\}\_\{t=1\}^\{N\}denote the dimension information and raw responses across all steps, respectively\.Mt−1=\{mj\}j=1t−1\{M\_\{t\-1\}\}=\\\{m\_\{j\}\\\}\_\{j=1\}^\{t\-1\}represents the accumulated memory of previous steps to preserve cross\-step dependency\.

### 3\.2Phase 2: Evidence\-based Judging

In this phase, Judge\-LLM performs scoring grounded in the structured evidence generated during Phase 1\. Rather than accessing the raw responsesRsR^\{s\}directly, Judge\-LLM takes the aggregated evidenceEE, the scoring rubricsRuR^\{u\}, andScenarioas input, and generates multi\-step and multi\-dimensional scoresSSover the entire response in a single pass, defined as:

S=fJ​u​d​g​e​\(S​c​e​n​a​r​i​o,E,D,Ru\)S=f\_\{Judge\}\(Scenario,E,D,R^\{u\}\)\(2\)
whereE=\{et\}t=1NE=\\\{e\_\{t\}\\\}\_\{t=1\}^\{N\}andRu=\{Rtu\}t=1NR^\{u\}=\\\{R^\{u\}\_\{t\}\\\}\_\{t=1\}^\{N\}denote the evidence extracted by SoT\-LLM and the scoring rubrics for all steps, respectively\.

As illustrated in Figure[1](https://arxiv.org/html/2609.03432#S2.F1), consider theAdequacydimension in Step\-2: the extracted evidence \(e\.g\., “low importance” and “a minor issue”\) indicates weak performance along this dimension, upon which Judge\-LLM assigns a relatively low score of 6 out of 10\. This evidence\-grounded judging enhances both the accuracy and stability of the final scoring\.

## 4Experiment

### 4\.1Challenging Dataset

We conduct experiments on the Contextually\-Grounded and Procedurally\-Structured Tasks \(CGPST\) dataset\([Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\), a complex multi\-step creativity benchmark\. It possesses a strong psychological foundation based on the Future Problem Solving Program International \(FPSPI\)\([Treffinger et al\., 2012b](https://arxiv.org/html/2609.03432#bib.bib33);[Alt et al\., 2022](https://arxiv.org/html/2609.03432#bib.bib34);[Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\), an active international creativity competition framework with over 50 years of history founded by psychologist Ellis Paul Torrance\([Torrance, 1966](https://arxiv.org/html/2609.03432#bib.bib4)\)\. Each task on CGPST is grounded in a future scenario and needs to complete six interdependent steps, each associated with multiple scoring dimensions, as shown in Table[1](https://arxiv.org/html/2609.03432#S1.T1)\. The dataset contains 10 different scenarios, each with 20 samples, resulting in a total of 200 samples\. Each sample includes complete responses of six steps and is annotated with calibrated scores from two human evaluators\. The inter\-rater reliability between the two human evaluators reaches 0\.84, indicating that the CGPST dataset is of high quality\. Due to its strong subjectivity, cross\-step dependencies, fine\-grained multi\-dimensional scoring, and wide score ranges, CGPST poses significant challenges for automated evaluation\. More details are shown in Appendix[A](https://arxiv.org/html/2609.03432#A1)\.

To further demonstrate the generalizability of CreaEval beyond complex multi\-step creativity task, we also evaluate our method on two classic creativity benchmarks with different task structures: \(1\)Alternative Uses Task \(AUT\)\([Organisciak et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib31);[Hadas and Hershkovitz, 2024](https://arxiv.org/html/2609.03432#bib.bib32)\), a widely used creativity test requiring participants to generate novel uses for a given object, scored exclusively onOriginality\. \(2\)Torrance Test of Creative Writing \(TTCW\)\([Chakrabarty et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib8)\), a narrative generation task requiring creative story writing based on a given plot, evaluated across four dimensions \(Fluency,Flexibility,Originality,Elaboration\)\. While we have incorporated AUT and TTCW dataset to ensure generalizability, CGPST remains our primary benchmark due to its comprehensive assessment and structural complexity\.

Table 2:Consistency \(QWK\) results across different methods and dimensions on the CGPST benchmark\.Boldindicates the highest score andunderlineindicates the second highest score\. The AVG column summarizes the overall average score\. The asterisk \(\*\) marks statistically significant improvements \(p < 0\.05, t\-test\) over the second\-best method\.
### 4\.2Baselines

We compare our CreaEval with the following baselines\. Detailed implementation details of all baselines are provided in Appendix[B](https://arxiv.org/html/2609.03432#A2)\.

- •Direct Score: This method directly assigns a score to a complete response without intermediate steps or decomposition\.
- •Chain\-of\-Thought\(CoT\)\([Wei et al\., 2022](https://arxiv.org/html/2609.03432#bib.bib12)\): This method decomposes the problem into intermediate steps and solve each before giving the final answer\.
- •Tree\-of\-Thought\(ToT\)\([Yao et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib13)\): This method actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving\.
- •Graph\-of\-Thought\(GoT\)\([Besta et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib14)\): This method models the problem\-solving process as a graph with more flexible thought transformations compared to ToT\.
- •Table as Thought\(TaT\)\([Sun et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib15)\): This method uses a table to represent structured thoughts\. Although this method shares the same structured representation, it does not decouple analysis and judging, making it anundecoupled variant of CreaEval\.
- •Supervised Fine\-Tuning\(SFT\): We fine\-tunesQwen3\.5\-9B111[https://huggingface\.co/Qwen/Qwen3\.5\-9B](https://huggingface.co/Qwen/Qwen3.5-9B)on a 7:3 train\-test split of the dataset, and reports results on test set\.
- •SaMer\([Feng et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib27)\): SaMer is a scenario\-aware multi\-dimensional LLM evaluator that adaptively identifies and weights dimensions according to different scenarios\.

Unlike CGPST, both AUT and TTCW are single\-turn generation tasks without procedural process and dependencies\. Consequently, we omit structure\-based baselines like ToT and GoT for these benchmarks\. Additionally, we additionally incorporate a reference\-based approach\([Li et al\., 2025](https://arxiv.org/html/2609.03432#bib.bib2)\)using human\-written stories as evaluation references as a train\-free baseline for TTCW\.

### 4\.3Experimental Setup

We employ four LLMs \(qwen3\.6\-plus,gpt\-5\.4,deepseek\-v4\-pro,gemini\-3\.1\-pro\) as Judge\-LLMs for all methods\. We useqwen3\.6\-plusas SoT\-LLM for structured evidence extraction\. Full LLM details are provided in Appendix[E](https://arxiv.org/html/2609.03432#A5)\. We report three agreement metrics: Pearson Correlation Coefficient \(PCC\), Quadratic Weighted Kappa \(QWK\), and Intra\-class Correlation Coefficient \(ICC\), with details in Appendix[C](https://arxiv.org/html/2609.03432#A3)\. Additionally, we set the temperature of all LLMs to 0\.2\. Appendix[F](https://arxiv.org/html/2609.03432#A6)shows a pilot study on temperature selection\. Appendix[D](https://arxiv.org/html/2609.03432#A4)presents the full prompts\.

Table 3:Consistency \(QWK\) results across different methods and dimensions on the AUT and TTCW benchmark\.Boldindicates the highest score andunderlineindicates the second highest score\. The AVG column summarizes the overall average score\. The asterisk \(\*\) marks statistically significant improvements \(p < 0\.05, t\-test\) over the second\-best method\.

## 5Results

### 5\.1Main Results

As shown in Table[2](https://arxiv.org/html/2609.03432#S4.T2)\(reporting QWK\. PCC and ICC show similar trends, see Appendix[H](https://arxiv.org/html/2609.03432#A8)for full results\), CreaEval consistently achieves the best performance \(0\.64\), significantly outperforming all baselines on the CGPST\. This indicates that CreaEval better aligns with human judgments with decoupled analysis and judging\. In contrast, other training\-free methods show relatively weak and similar performance \(0\.2\-0\.25\), suggesting that simply increasing reasoning steps does not improve accuracy for subjective dimensions evaluation\. For training\-based methods, SFT achieves the second\-best performance \(0\.47\), indicating that supervised data can effectively align LLM judging with human preferences to a certain extent, but still falls short of CreaEval\. Furthermore, as presented in Table[3](https://arxiv.org/html/2609.03432#S4.T3), CreaEval also achieves superior performance on simple creativity tasks, demonstrating its strong generalizability to simpler creativity tasks beyond complex multi\-step evaluations\.

Performance varies across dimensions\. As shown in Table[2](https://arxiv.org/html/2609.03432#S4.T2), for dimensions with large range such as Step\-2 \(Focus,Adequacy\) and Step\-4 \(Relevance\), most training\-free baselines remain below 0\.2, indicating that they struggle to accurately distinguish score differences within wide ranges\. In contrast, Step\-5 is relatively less subjective, as it mainly involves verifying whether scoring vectors satisfy a non\-repetitive ranking structure\. Nevertheless, most training\-free methods still achieve only weak performance \(around 0\.08–0\.46 v\.s\. CreaEval 0\.94\), as they retain redundant information from previous steps without effective evidence extraction, which further disrupts subsequent evaluations\. Step\-6 is the most challenging step, requiring holistic integration across all previous steps to produce a final action plan\. In Step\-6, all training\-free baselines collapse to very weak performance \(<0\.2\), whereas CreaEval still maintains moderate consistency \(around 0\.5\)\. This further validates the effectiveness of the decoupled analysis\-judging design\.

Furthermore, as shown in Figure[2](https://arxiv.org/html/2609.03432#S5.F2), we analyze the consistency of four Judge\-LLMs under each method\. CreaEval exhibits nearly identical performance across all judges, indicating stable and robust evaluation behavior\. This stability stems from replacing raw responses with structured evidence, which reduces the impact of subjective variation in the raw responses in scoring, further demonstrating that CreaEval is a model\-agnostic framework that does not rely on the specific Judge\-LLM\. In contrast, training\-free methods show noticeable fluctuations across different judges due to variations in model capability, resulting in more diverse scoring behavior\. We further provide an in\-depth analysis of scoring stability in Section[6\.3](https://arxiv.org/html/2609.03432#S6.SS3)\.

Figure 2:Judge QWK Consistency Across Methods\.
### 5\.2Ablation Study

We conduct ablation studies to evaluate the effectiveness of key components in CreaEval\. Specifically, we consider four variants: \(1\)w/o Memory, where the memory mechanism in Phase 1 is removed, and each step is processed independently without cross\-step dependency; \(2\)w/o Evidence, where Judge\-LLM directly performs scoring based on the raw responses without extracting structured evidence, corresponding to Direct Score baseline; \(3\)w/o Decoupled, where analysis and judging are performed jointly within a single LLM without separation, corresponding to TaT baseline; and \(4\)w/o Step\-wise, where evidence extraction is performed in a single pass over all six steps instead of step\-by\-step iterative extraction\.

Table[4](https://arxiv.org/html/2609.03432#S5.T4)presents the results\. Removing memory leads to performance degradation \(QWK = 0\.49, 23%↓\\downarrow\), demonstrating that the memory mechanism is necessary for maintaining cross\-step coherence\. Without evidence, the performance decreases \(QWK = 0\.24, 63%↓\\downarrow\), indicating that evidence\-based scoring is more accurate than direct scoring\. Similarly, when analysis and judging are not decoupled, performance further degrades \(QWK = 0\.21, 67%↓\\downarrow\), suggesting that coupling the two processes introduces analytical bias into scoring\. Meanwhile, extracting evidence for all steps at once leads to a collapse in performance \(QWK = 0\.01, 98%↓\\downarrow\), indicating that excessive input information significantly harms SoT\-LLM’s evidence extraction accuracy\. For example, SoT\-LLM provides evidence for some dimensions but only produces binary judgments \(e\.g\.,“Yes”or“No”\) for others, which leads to significant degradation in scoring quality\.

Table 4:Ablation results\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-3_Originality.png)Figure 3:Headmap between human score and all methods onOriginalityof Step\-3\.

## 6Discussion

In this section, we conduct a comprehensive analysis of CreaEval, including robustness, bias mitigation, scoring stability, efficiency, and case study, further validating the effectiveness of CreaEval beyond simple improvements in scoring accuracy\.

### 6\.1Impact of SoT\-LLM in CreaEval

The quality of extracted evidence directly affects the accuracy of the final scoring, making SoT\-LLM a critical component\. To evaluate the robustness of CreaEval, we replace SoT\-LLM with other LLMs and report QWK across different LLMs combinations\. As shown in Figure[4](https://arxiv.org/html/2609.03432#S6.F4), the agreement remains consistently above 0\.6 across all SoT\-LLMs and Judge\-LLMs combinations, indicating that CreaEval does not rely on a specific LLM and can serve as a general and robust evaluation framework\. PCC and ICC results are provided in Appendix[I](https://arxiv.org/html/2609.03432#A9)\.

Figure 4:QWK results across different SoT\-LLMs\.
### 6\.2Bias Mitigation in CreaEval

Figure 5:Relative changes in PCC against Direct Score\. PCC measures the linear correlation between Step\-6 response length andDevelopmentscores\.In LLM\-as\-a\-Judge, LLMs are prone to various evaluation biases during scoring\. In automated evaluation of creative tasks, two biases are particularly critical:\(1\) leniency bias, where LLMs tend to assign overly high scores to subjective dimensions due to their inherent sycophantic tendencies\([Ye et al\., 2025a](https://arxiv.org/html/2609.03432#bib.bib24);[Gupta et al\., 2026](https://arxiv.org/html/2609.03432#bib.bib23)\); and\(2\) verbosity bias, where LLM favor longer responses over shorter ones, even when the latter are clearer or of higher quality\([Zheng et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib1)\)\.

For leniency bias, we analyze the heatmap between different methods and human scores onOriginalityof Step\-3, as shown in Figure[3](https://arxiv.org/html/2609.03432#S5.F3)\. The results indicate that all training\-free methods exhibit a clear inclination to cluster their scores within the high\-value region\. In contrast, CreaEval significantly mitigates this tendency, demonstrating a more balanced score distribution that aligns closely with human annotations\. Heatmaps for additional dimensions are provided in Appendix[J](https://arxiv.org/html/2609.03432#A10)\.

For verbosity bias, we examine PCC between response length of Step\-6 andDevelopment\(the level of detail of action plan\) scores to measure their linear correlation\. Figure[5](https://arxiv.org/html/2609.03432#S6.F5)illustrates the relative changes for training\-free methods against Direct Score\. Only CreaEval shows a decrease \(\-5\.81%\), indicating that response length is less correlated with scores compared to Direct Score and thus verbosity bias is mitigated\. However, CoT, GoT, and ToT even exacerbate the bias despite introducing more complex reasoning paths\. TaT shows a slight increase \(\+0\.82%\), suggesting that incorporating analysis can partially reduce verbosity bias\. However, due to its coupled analysis\-and\-judging design, its effectiveness remains inferior to CreaEval\.

Figure 6:Distribution of inter\-Judge variance across the entire dataset forCorrectly Usedin Step\-5\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step2.png)Figure 7:Case study of Step\-2 comparing Table as Thought \(TaT\) and CreaEval\. The example illustrates that TaT tends to produce overly positive and less discriminative judgments \(yellow\) due to coupled analysis and scoring within a single LLM, leading to leniency bias\. In contrast, CreaEval separates evidence extraction and judging into distinct LLMs, enabling more grounded and constraint\-aware judgments \(blue\)\.
### 6\.3Scoring Stability Analysis

We analyze scoring stability by measuring the distribution of inter\-Judge variance for each method\. Specifically, for each sample and dimension, we compute the variance of scores assigned by four Judge\-LLMs\. This yields 200 variance values per method, corresponding to all samples on CGPST dataset, which are then analyzed as a distribution\. Figure[6](https://arxiv.org/html/2609.03432#S6.F6)illustrates the variance distribution forCorrectly Usedin Step\-5\. CreaEval exhibits consistently lower inter\-Judge variance, indicating the four Judge\-LLMs produce highly similar scores, leading to higher scoring stability\. This is attributed to its evidence\-based judging design, where all Judges rely on shared extracted evidence rather than raw subjective responses, leading to more consistent evaluations\. In contrast, other methods show substantially higher and more dispersed variance, suggesting unstable judgments across different Judges even for the same response\. Violin plots for other dimensions are provided in Appendix[K](https://arxiv.org/html/2609.03432#A11)\.

### 6\.4Inference Cost Analysis

Table 5:Time and token consumption\. Both are reported in terms of the 50th and 95th percentile \(p50/p95\)\.We further analyze the inference cost of different methods in terms of time and token consumption, as shown in Table[5](https://arxiv.org/html/2609.03432#S6.T5)\. Overall, methods with more complex reasoning exhibit significantly higher consumption\. In particular, GoT and ToT introduce substantial overhead due to expanded reasoning paths\. In contrast, CoT and TaT achieves relatively low consumption\. CreaEval maintains a better balance, achieving substantially lower cost than ToT and GoT while maintaining strong performance\.

### 6\.5Case Study

As shown in Figure[7](https://arxiv.org/html/2609.03432#S6.F7), we present a case study of Step\-2\. TaT, an undecoupled variant of CreaEval, tends to produce overly positive evaluations \(e\.g\., “a high level of focus”\), exhibiting clear leniency bias\. In contrast, CreaEval decouples evidence extraction from judging, grounding scoring in structured intermediate evidence rather than raw responses\. This design mitigates over\-optimistic scoring \(e\.g\., “less focused problem”\), leading to more accurate evaluations\. Case studies for other steps are provided in Appendix[L](https://arxiv.org/html/2609.03432#A12)\.

## 7Conclusions

In this work, we propose CreaEval, an automated creativity evaluation framework for CGPST that decouples evaluation into memory\-augmented analysis and evidence\-based judging\. SoT\-LLM first converts multi\-step responses into structured evidence with cross\-step memory, and Judge\-LLM performs scoring based solely on this evidence without accessing the raw responses\. Experiments across four LLMs show that CreaEval achieves strong alignment with human judgments, consistently outperforming all baselines\. Further analysis demonstrates that the decoupled design improves scoring stability and reduces verbosity and leniency biases in LLM\-as\-a\-Judge, offering new insights into evaluating subjective creativity tasks\.

## Limitations

Although our work demonstrates strong effectiveness and achieves promising results, it still has several limitations\. In Phase 1, we use a simple rule\-based memory mechanism to maintain cross\-step coherence due to the fixed step dependencies on CGPST\. More advanced memory modules such as hierarchical memory modules could be explored for more general settings\. In addition, to the best of our knowledge, CGPST is currently the only publicly available multi\-step creativity benchmark dataset, so our experiments are conducted exclusively on this multi\-step dataset\. Future work will evaluate the effectiveness of CreaEval on additional datasets once they become available\.

## References

- Altet al\.\(2022\)D\. Alt, Y\. Kapshuk, and H\. DekelPromoting perceived creativity and innovative behavior: benefits of future problem\-solving programs for higher education students\.Thinking Skills and Creativity47,pp\. 101201\.External Links:[Document](https://dx.doi.org/10.1016/j.tsc.2022.101201)Cited by:[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p1.1)\.
- Bestaet al\.\(2024\)M\. Besta, N\. Blach, A\. Kubicek, R\. Gerstenberger, M\. Podstawski, L\. Gianinazzi, J\. Gajda, T\. Lehmann, H\. Niewiadomski, P\. Nyczyk, and T\. HoeflerGraph of thoughts: solving elaborate problems with large language models\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 17682–17690\.External Links:ISSN 2159\-5399,[Link](http://dx.doi.org/10.1609/aaai.v38i16.29720),[Document](https://dx.doi.org/10.1609/aaai.v38i16.29720)Cited by:[Appendix B](https://arxiv.org/html/2609.03432#A2.SS0.SSS0.Px4.p1.1),[4th item](https://arxiv.org/html/2609.03432#S4.I1.i4.p1.1)\.
- Chakrabartyet al\.\(2024\)T\. Chakrabarty, P\. Laban, D\. Agarwal, S\. Muresan, and C\. WuArt or artifice? large language models and the false promise of creativity\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,CHI ’24,New York, NY, USA\.External Links:ISBN 9798400703300,[Link](https://doi.org/10.1145/3613904.3642731),[Document](https://dx.doi.org/10.1145/3613904.3642731)Cited by:[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p2.1)\.
- Doet al\.\(2024\)H\. Do, Y\. Kim, and G\. LeeAutoregressive score generation for multi\-trait essay scoring\.InFindings of the Association for Computational Linguistics: EACL 2024,Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 1659–1666\.External Links:[Link](https://aclanthology.org/2024.findings-eacl.115/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-eacl.115)Cited by:[Appendix C](https://arxiv.org/html/2609.03432#A3.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.03432#S1.p3.1)\.
- Fenget al\.\(2025\)K\. Feng, K\. Ding, J\. Yu, Y\. Qu, Z\. Chen, c\. lv, G\. Yu, Q\. Zhang, and H\. ChenSaMer: a scenario\-aware multi\-dimensional evaluator for large language models\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 40346–40367\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/646ca7b994bc46afe33d680dbe7ed67a-Paper-Conference.pdf)Cited by:[7th item](https://arxiv.org/html/2609.03432#S4.I1.i7.p1.1)\.
- Guptaet al\.\(2026\)M\. Gupta, I\. Nair, L\. Wang, and D\. KumarContext over content: exposing evaluation faking in automated judges\.External Links:2604\.15224,[Link](https://arxiv.org/abs/2604.15224)Cited by:[§6\.2](https://arxiv.org/html/2609.03432#S6.SS2.p1.1)\.
- Hadas and Hershkovitz \(2024\)E\. Hadas and A\. HershkovitzUsing large language models to evaluate alternative uses task flexibility score\.Thinking Skills and Creativity52,pp\. 101549\.External Links:ISSN 1871\-1871,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.tsc.2024.101549),[Link](https://www.sciencedirect.com/science/article/pii/S1871187124000877)Cited by:[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p2.1)\.
- Harsch and Martin \(2013\)C\. Harsch and G\. MartinComparing holistic and analytic scoring methods: issues of validity and reliability\.Assessment in Education: Principles, Policy & Practice20\(3\),pp\. 281–307\.Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p5.1)\.
- Kleinet al\.\(1998\)S\. P\. Klein, B\. M\. Stecher, R\. J\. Shavelson, D\. McCaffrey, T\. Ormseth, R\. M\. Bell, K\. Comfort, and A\. R\. OthmanAnalytic versus holistic scoring of science performance tasks\.Applied Measurement in Education11\(2\),pp\. 121–137\.Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p5.1)\.
- Kumaret al\.\(2025\)H\. Kumar, J\. Vincentius, E\. Jordan, and A\. AndersonHuman creativity in the age of llms: randomized experiments on divergent and convergent thinking\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems,CHI ’25,New York, NY, USA\.External Links:ISBN 9798400713941,[Link](https://doi.org/10.1145/3706598.3714198),[Document](https://dx.doi.org/10.1145/3706598.3714198)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1)\.
- Liet al\.\(2025\)R\. Li, C\. Zhu, B\. Xu, X\. Wang, and Z\. MaoAutomated creativity evaluation for large language models: a reference\-based approach\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 21475–21488\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1171/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1171),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2609.03432#S4.SS2.p3.1)\.
- Li and Pan \(2025\)X\. Li and W\. PanKAES: multi\-aspect shared knowledge finding and aligning for cross\-prompt automated scoring of essay traits\.Proceedings of the AAAI Conference on Artificial Intelligence39\(23\),pp\. 24476–24484\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34626),[Document](https://dx.doi.org/10.1609/aaai.v39i23.34626)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p3.1)\.
- Lianget al\.\(2024\)T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. TuEncouraging divergent thinking in large language models through multi\-agent debate\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17889–17904\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.992/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.992)Cited by:[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1)\.
- Linet al\.\(2025\)Y\. Lin, K\. Chen, Z\. Li, T\. Wu, T\. Wu, K\. Chen, H\. Lee, and Y\. ChenCreativity in LLM\-based multi\-agent systems: a survey\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27584–27607\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1403/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1403),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p1.1)\.
- Luet al\.\(2024\)L\. Lu, S\. Chen, T\. Pai, C\. Yu, H\. Lee, and S\. SunLLM discussion: enhancing the creativity of large language models via discussion framework and role\-play\.External Links:2405\.06373,[Link](https://arxiv.org/abs/2405.06373)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1)\.
- Organisciaket al\.\(2023\)P\. Organisciak, S\. Acar, D\. Dumas, and K\. BerthiaumeBeyond semantic distance: automated scoring of divergent thinking greatly improves with large language models\.Thinking Skills and Creativity49,pp\. 101356\.External Links:ISSN 1871\-1871,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.tsc.2023.101356),[Link](https://www.sciencedirect.com/science/article/pii/S1871187123001256)Cited by:[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p2.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,P\. Isabelle, E\. Charniak, and D\. Lin \(Eds\.\),Philadelphia, Pennsylvania, USA,pp\. 311–318\.External Links:[Link](https://aclanthology.org/P02-1040/),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[Appendix F](https://arxiv.org/html/2609.03432#A6.p2.1)\.
- Qiet al\.\(2025\)R\. Qi, Z\. Man, Y\. Chen, F\. Mo, J\. Xu, and K\. HuangSoT: structured\-of\-thought prompting guides multilingual reasoning in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11024–11039\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.586/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.586),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p5.1)\.
- Ridleyet al\.\(2021\)R\. Ridley, L\. He, X\. Dai, S\. Huang, and J\. ChenAutomated cross\-prompt scoring of essay traits\.Proceedings of the AAAI Conference on Artificial Intelligence35\(15\),pp\. 13745–13753\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/17620),[Document](https://dx.doi.org/10.1609/aaai.v35i15.17620)Cited by:[Appendix C](https://arxiv.org/html/2609.03432#A3.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2025\)Z\. Sun, N\. Deng, H\. Yu, and J\. YouTables as thought: exploring structured thoughts in LLM reasoning\.InProceedings of the 4th Table Representation Learning Workshop,S\. Chang, M\. Hulsebos, Q\. Liu, W\. Chen, and H\. Sun \(Eds\.\),Vienna, Austria,pp\. 19–33\.External Links:[Link](https://aclanthology.org/2025.trl-1.3/),[Document](https://dx.doi.org/10.18653/v1/2025.trl-1.3),ISBN 979\-8\-89176\-268\-8Cited by:[5th item](https://arxiv.org/html/2609.03432#S4.I1.i5.p1.1)\.
- Torrance \(1966\)E\. P\. TorranceTorrance tests of creative thinking\.Educational and psychological measurement\.Cited by:[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p1.1)\.
- Treffingeret al\.\(2012a\)D\. J\. Treffinger, M\. Solomon, and D\. WoythalFour decades of creative vision: insights from an evaluation of the future problem solving program international \(fpspi\)\.The Journal of Creative Behavior46\(3\),pp\. 209–219\.Cited by:[Appendix A](https://arxiv.org/html/2609.03432#A1.p1.1),[§1](https://arxiv.org/html/2609.03432#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1)\.
- Treffinger \(1995\)D\. J\. TreffingerCreative problem solving: overview and educational implications\.Educational Psychology Review7\(3\),pp\. 301–312\.External Links:ISSN 1573\-336X,[Link](http://dx.doi.org/10.1007/BF02213375),[Document](https://dx.doi.org/10.1007/bf02213375)Cited by:[Appendix A](https://arxiv.org/html/2609.03432#A1.p1.1),[§1](https://arxiv.org/html/2609.03432#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1)\.
- Treffingeret al\.\(2012b\)D\. Treffinger, M\. Solomon, and D\. WoythalFour decades of creative vision: insights from an evaluation of the future problem solving program international \(fpspi\)\.The Journal of Creative Behavior46,pp\.\.External Links:[Document](https://dx.doi.org/10.1002/jocb.14)Cited by:[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p1.1)\.
- Wang and Liu \(2025\)J\. Wang and J\. LiuT\-MES: trait\-aware mix\-of\-experts representation learning for multi\-trait essay scoring\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 1224–1236\.External Links:[Link](https://aclanthology.org/2025.coling-main.81/)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p3.1)\.
- Wanget al\.\(2026a\)Q\. Wang, H\. Ye, J\. Kim, J\. Ke, Y\. Wang, M\. Kuo, Z\. Shao, D\. Li, Y\. Lin, T\. Jiang, C\. Wei, Q\. Qian, W\. Wen, H\. Li, and Y\. ChenT2S\-bench & structure\-of\-thought: benchmarking and prompting comprehensive text\-to\-structure reasoning\.External Links:2603\.03790,[Link](https://arxiv.org/abs/2603.03790)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p5.1)\.
- Wanget al\.\(2026b\)X\. Wang, J\. Wu, H\. Shi, W\. Xia, J\. Yu, and C\. ZhengTeamLLM: a human\-like team\-oriented collaboration framework for multi\-step contextualized tasks\.External Links:2604\.06765,[Link](https://arxiv.org/abs/2604.06765)Cited by:[Appendix A](https://arxiv.org/html/2609.03432#A1.p1.1),[§1](https://arxiv.org/html/2609.03432#S1.p2.1),[§1](https://arxiv.org/html/2609.03432#S1.p3.1),[§1](https://arxiv.org/html/2609.03432#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.03432#S4.SS1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, b\. ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)Cited by:[Appendix B](https://arxiv.org/html/2609.03432#A2.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2609.03432#S4.I1.i2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 11809–11822\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf)Cited by:[Appendix B](https://arxiv.org/html/2609.03432#A2.SS0.SSS0.Px3.p1.1),[3rd item](https://arxiv.org/html/2609.03432#S4.I1.i3.p1.1)\.
- Yeet al\.\(2025a\)J\. Ye, Y\. Wang, Y\. Huang, D\. Chen, Q\. Zhang, N\. Moniz, T\. Gao, W\. Geyer, C\. Huang, P\. Chen, N\. Chawla, and X\. ZhangJustice or prejudice? quantifying biases in llm\-as\-a\-judge\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 102351–102390\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/fdca08d371e4b6c031397909e20043bd-Paper-Conference.pdf)Cited by:[§6\.2](https://arxiv.org/html/2609.03432#S6.SS2.p1.1)\.
- Yeet al\.\(2025b\)J\. Ye, J\. Gu, X\. Zhao, W\. Yin, and G\. WangAssessing the creativity of llms in proposing novel solutions to mathematical problems\.Proceedings of the AAAI Conference on Artificial Intelligence39\(24\),pp\. 25687–25696\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34760),[Document](https://dx.doi.org/10.1609/aaai.v39i24.34760)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p1.1),[§1](https://arxiv.org/html/2609.03432#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with bert\.External Links:1904\.09675,[Link](https://arxiv.org/abs/1904.09675)Cited by:[Appendix F](https://arxiv.org/html/2609.03432#A6.p2.1)\.
- Zhaoet al\.\(2025\)Y\. Zhao, R\. Zhang, W\. Li, and L\. LiAssessing and understanding creativity in large language models\.Machine Intelligence Research22\(3\),pp\. 417–436\.External Links:ISSN 2731\-5398,[Link](http://dx.doi.org/10.1007/s11633-025-1546-4),[Document](https://dx.doi.org/10.1007/s11633-025-1546-4)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.03432#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46595–46623\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)Cited by:[§1](https://arxiv.org/html/2609.03432#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.03432#S2.SS2.p1.1),[§6\.2](https://arxiv.org/html/2609.03432#S6.SS2.p1.1)\.

## Appendix ACGPST Benchmark Details

Contextually\-Grounded and Procedurally\-Structured tasks \(CGPST\)\([Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\)is a benchmark designed to evaluate the comprehensive problem\-solving abilities of LLMs through contextualized multi\-step tasks\. Compared with traditional creativity benchmarks such as AUT and TTCT, CGPST is substantially more complex and therefore more challenging for automated evaluation\. Specifically, CGPST exhibits four key characteristics:Contextual Grounding, where each task is based on a complete future scenario with rich contextual information and detailed problem settings;Procedural Structure, where each task consists of six interdependent and sequential steps progressively leading to the resolution of a real\-world problem;Process\-Oriented Evaluation, where all intermediate steps are systematically evaluated rather than focusing only on the final response; andMulti\-Dimensional Assessment, where each step is evaluated across multiple comprehensive dimensions\([Treffinger, 1995](https://arxiv.org/html/2609.03432#bib.bib19);[Treffinger et al\., 2012a](https://arxiv.org/html/2609.03432#bib.bib18);[Wang et al\., 2026b](https://arxiv.org/html/2609.03432#bib.bib11)\)\.

The CGPST dataset contains 10 different scenarios, each consisting of 20 complete six\-step response samples, resulting in a total of 200 samples\. Each sample is annotated with validated scores from two human experts\. To facilitate understanding, we provide a complete future scenario inOcean Soup Future Scenariobox \(Page[C](https://arxiv.org/html/2609.03432#A3.SS0.SSS0.Px3)\) and an example sample inA Complete CGPST Samplebox \(Page[C](https://arxiv.org/html/2609.03432#A3.SS0.SSS0.Px3)\)\. Table[6](https://arxiv.org/html/2609.03432#A1.T6)presents the themes of all scenarios\.

As shown in Table[1](https://arxiv.org/html/2609.03432#S1.T1), CGPST consists of six sequential steps\. Given a future scenario, LLMs are first required to identify up to eight challenges \(Step\-1\), then select the most promising challenge as an underlying problem \(Step\-2\), and generate up to eight solutions for this problem \(Step\-3\)\. Subsequently, LLMs create five evaluation criteria for the problem and proposed solutions \(Step\-4\), which are then used to rank the solutions and select the best one \(Step\-5\)\. Finally, the selected solution is further developed into a comprehensive action plan addressing the underlying problem and generating positive impacts on the future scenario \(Step\-6\)\. The strong interdependency across all six steps further increases the difficulty of automated evaluation\. We provide detailed descriptions of dimensions for each step in Table[14](https://arxiv.org/html/2609.03432#A12.T14)to[19](https://arxiv.org/html/2609.03432#A12.T19)\.

Table 6:Overview of the 10 future scenarios on the CGPST dataset\. Each scenario contains 20 six\-step response samples, resulting in a total of 200 samples annotated with two validated human expert scores\.
## Appendix BBaseline Implementation Details

We provide the detailed implementation of the baseline methods as follows\.

#### Direct Score

LLM directly assigns scores to all steps of a response in a single pass\. The input includes the future scenario, the full response, dimension descriptions, and corresponding rubrics\.

#### Chain\-of\-Thought \(CoT\)

Given the inherent multi\-step structure of CGPST, the evaluation is performed step by step, where each step is scored sequentially\. Specifically, a single response requires six sequential passes corresponding to the six steps\([Wei et al\., 2022](https://arxiv.org/html/2609.03432#bib.bib12)\)\.

#### Tree\-of\-Thought \(ToT\)

ToT follows a tree\-structured evaluation process, with each layer aligned to a step on CGPST\. At each layer, three independent Score LLMs generate three candidate scores, forming three branches of this layer\. A separate Judge LLM then selects the most appropriate score among these candidates as the final output for this step\([Yao et al\., 2023](https://arxiv.org/html/2609.03432#bib.bib13)\)\.

#### Graph\-of\-Thought \(GoT\)

Built upon ToT, GoT introduces an additional refinement operation for thought transformation\. Specifically, for each step, three Score LLMs first generate candidate scores, which are then refined through a self\-refinement process before final selection\. The Judge LLM then selects the final score from the refined candidates for each step\. This refinement mechanism allows iterative improvement of intermediate scoring nodes, forming a graph\-structured reasoning process\([Besta et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib14)\)\.

#### Table as Thought

This method organizes reasoning within a tabular schema\. In the original paper, the table is stored in JSON format, which is consistent with the representation format used in CreaEval\. However, unlike CreaEval, both analysis and judging are performed by a single LLM within this framework, making it anundecoupled variant of CreaEval\. We therefore regard it as a baseline that shares the same structured representation but does not decouple the evaluation process\.

#### Supervised Fine\-Tuning \(SFT\)

We split the CGPST dataset into training and test sets with a ratio of 7:3\. Subsequently, we perform SFT onQwen3\.5\-9Busing the instruction\-response pairs constructed from the training set\. After training, the model’s capability is evaluated on the test set\.

#### Scenario\-aware Multi\-dimensional Evaluator \(SaMer\)

SaMer introduces a three\-branch scoring head atop a frozenQwen3\.5\-9Bto enable scenario\-aware, preference\-driven evaluation\. During training, the base LLM is frozen and only the head is optimized through a multi\-objective loss combining three signals: a dimension prediction loss that learns step\-to\-dimension relevance, a dimension\-level ranking loss that aligns pairwise preferences on individual dimensions, and an overall preference loss that learns the final winner from pairwise comparisons\. At inference, only dimensions predicted as relevant receive non\-zero weights via softmax, and the overall score is computed as the weighted sum of dimension scores\.

## Appendix CAgreement Metric Details

On the CGPST dataset, each sample is annotated by two human raters\. Therefore, when computing agreement between LLM scores and human annotations,we calculate the consistency separately with Human A and Human B for each dimension, and then average the two results as the final agreement score\. Importantly, we do not compute agreement against the averaged human score\. This is to avoid misleading cases where Human A gives a score of 4 and Human B gives 8; an LLM score of 6 would appear perfectly accurate if compared to the mean of two humans \(6\), even though it does not truly match either annotator\.

We present three metrics below to facilitate a better understanding of the experimental results\.

#### Pearson correlation coefficient \(PCC\)

PCC measures thelinear correlationbetween predicted scores and human annotations\. It evaluates how well the model predictions align with human judgments in terms of overall trend consistency\.

P​C​C=∑i=1n\(xi−x¯\)​\(yi−y¯\)∑i=1n\(xi−x¯\)2​∑i=1n\(yi−y¯\)2PCC=\\frac\{\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\bar\{x\}\)\(y\_\{i\}\-\\bar\{y\}\)\}\{\\sqrt\{\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\bar\{x\}\)^\{2\}\}\\sqrt\{\\sum\_\{i=1\}^\{n\}\(y\_\{i\}\-\\bar\{y\}\)^\{2\}\}\}\(3\)
wherexix\_\{i\}andyiy\_\{i\}denote the predicted and human scores for sampleii, andx¯\\bar\{x\}andy¯\\bar\{y\}denote their respective means\.

#### Quadratic weighted kappa \(QWK\)

QWK measures the agreement between two raters while taking into account theordinal natureof the ratings and the degree of disagreement\. It is particularly suitable for discrete or ordinal scoring tasks and is one of the most commonly used metrics in automated scoring\([Ridley et al\., 2021](https://arxiv.org/html/2609.03432#bib.bib28);[Do et al\., 2024](https://arxiv.org/html/2609.03432#bib.bib20)\)\.

Q​W​K=1−∑i,jwi​j​Oi​j∑i,jwi​j​Ei​jQWK=1\-\\frac\{\\sum\_\{i,j\}w\_\{ij\}O\_\{ij\}\}\{\\sum\_\{i,j\}w\_\{ij\}E\_\{ij\}\}\(4\)wi​j=\(i−j\)2\(N−1\)2w\_\{ij\}=\\frac\{\(i\-j\)^\{2\}\}\{\(N\-1\)^\{2\}\}\(5\)
whereOi​jO\_\{ij\}andEi​jE\_\{ij\}are the observed and expected agreement matrices, respectively, andwi​jw\_\{ij\}is the quadratic weight\.

#### Intra\-class correlation coefficient \(ICC\)

ICC is widely used to assess inter\-rater reliability by measuring the proportion of variance attributable to differences between subjects relative to total variance\. It reflects the consistency of quantitative measurements across different raters\.

I​C​C=σbetween2σbetween2\+σwithin2ICC=\\frac\{\\sigma^\{2\}\_\{\\text\{between\}\}\}\{\\sigma^\{2\}\_\{\\text\{between\}\}\+\\sigma^\{2\}\_\{\\text\{within\}\}\}\(6\)
whereσbetween2\\sigma^\{2\}\_\{\\text\{between\}\}andσwithin2\\sigma^\{2\}\_\{\\text\{within\}\}denote the between\-subject and within\-subject variance, respectively\.

Ocean Soup Future ScenarioAs Jobie Sakai leans on the railing of the Ola Kai, she admires the beauty of her ocean paradise\. Jobie, a fifth generation Hawaiian devoted to the future of her homeland, is dedicated to her job as an environmental chemist aboard the Ola Kai \#6, one of Hawai’i’s floating science laboratories\. The Ola Kai \(meaning "healthy ocean"\) Project is a combined effort of the Hawaiian Environmental Council and the University of Hawai’i\.Now, in 2035 after 15 active years, the project is struggling to live up to its nickname: the OK Project\. Originally, the OK Project focused on the waters affected by Hawai’i’s island\-generated pollutants\. Public interest in the project led to a resurgence in eco\-education; recycling and the reduced use of plastics became an accepted part of island life for Hawai’i residents\. The "adopt a beach" clean\-up program became a popular draw for eco\-tourists\. However, researchers like Jobie became increasingly aware that their efforts were not enough\.The world’s largest manufacturers of plastic products border both sides of the Pacific\. A ten million square mile system of rotating currents called the North Pacific Gyre has its axis near the 137 islands of the Hawaiian chain\. Pacific environmental regulations have historically been weak or disregarded by heavy industrial nations who continue to use these waters as a dumping ground\. Consequently, the 1500 mile\-long archipelago paradise has been attacked by ocean soup for many years\.The soup surrounds Hawai’i, placing the islands and their resources at risk of permanent damage\. Especially vulnerable are the sparsely\-inhabited northwest islands, the world’s largest protected marine sanctuary that is home to many endangered fish, birds, seals, and Hawai’i’s beleaguered fishing industry\. Eco\-tourism routes have been altered to reduce impact on indigenous species and circumnavigated due to the location of floating laboratories\.With the increasing damaging effects from ocean soup on the island chain, Jobie and her coworkers realized that the Ola Kai Project’s numerous floating labs had to broaden their territory while narrowing their focus\. Due to the scope of the damage, the project directors reached out to other groups working in the Pacific and consulted with the National Oceanic and Atmospheric Administration \(NOAA\)\. It was determined that the best approach would be to divide up the responsibilities among agencies\. Now the OK Project labs focus solely on the battle against microplastics, leaving the collection of larger trash to other organizations\.Ola Kai laboratory crews record various data for analysis, keeping track of multiple fish species and beneficial organisms like plankton\. Specified lab crews weigh the microplastic debris collected on a weekly basis and compile that data while Jobie and other chemists continue their in\-depth examination of the plastic degradation and its effect on the waters surrounding the islands\.Collection of debris that is smaller than a pencil eraser has often done more harm than good to sea life\. After experimenting with several collection methods, the OK Project currently uses below\-surface robotic collectors that randomly collect plastic particles being carried by the currents\. Project teams are also experimenting with alternative collection methods, including new nanofiber sieves and use of pollution\-dissolving lasers\.Ola Kai’s floating labs have plastic\-to\-fuel conversion systems capable of harvesting tons of plastic pollution and converting it into diesel fuel for the labs, thus eliminating the need to return to shore for disposal of the waste in a landfill\. In spite of their progress, water samples still show an alarming amount of plastic particles\.

A Complete CGPST Sample\#\#\#Step\-1 Identify Challenges1\. The concentration of microplastics may have exceeded the density of plankton by tenfold, disrupting the energy input of the base food chain\. This challenge arises from the warning in the scenario that "water samples still show an alarming amount of plastic particles\."2\. Subsurface robotic collectors may miss low\-velocity eddy zones, creating data gaps and masking local ecological collapse points\. This is directly related to the scenario’s mention that "after experimenting with several collection methods," weekly weighing is still required, implying sampling limitations\.3\. The plastic\-to\-fuel conversion system may emit nanoscale black carbon particles, which could exacerbate imbalances in ocean surface heat absorption\. This challenge stems from the scenario emphasizing "eliminating the need to return to shore for disposal" without addressing potential secondary emissions\.4\. Endangered species in the northwest islands may face unknown toxicological effects from ingesting micro\-fragments of plastics broken down by lasers\. The scenario mentions that laser technology "can dissolve pollutants" but does not evaluate the byproducts of fragmentation\.5\. Adjustments to eco\-tourism routes around floating laboratories may transfer visitor pressure to other more fragile reefs\. This challenge is directly related to the scenario’s note that routes were "altered to reduce impact" but without ensuring overall pressure reduction\.6\. Legal exemptions for manufacturers on both sides of the Pacific may cause Hawaiian regional governance to operate in isolation\. This challenge is closely related to the scenario’s statement that "environmental regulations have historically been weak or disregarded\."7\. The efficiency of the plastic\-to\-diesel system may suddenly decline due to sea spray corrosion, forcing laboratories to rely on land\-based resupply\. This challenge is implied in the scenario mentioning "harvesting tons of plastic" without considering long\-term durability\.8\. Data protocols among the network of floating labs may be incompatible, hindering multinational collaboration in compiling a comprehensive microplastic hotspot map\. This challenge is directly related to the scenario emphasizing "dividing responsibilities among agencies" without a unified standard\.\#\#\#Step\-2 Select an Underlying ProblemChallenge ID: 1\. In 2035, in the waters surrounding the Hawaiian Islands, when microplastic concentrations may have exceeded the density of plankton tenfold, threatening the energy input at the base of the food chain, how can we reduce the overwhelming proportion of microplastics at the bottom of the food chain to restore energy flow in the Pacific ecosystem and protect the future of endangered species?Time: 2035Location: Waters surrounding the Hawaiian IslandsTheme: Microplastic Management and Ecological Restoration\#\#\#Step\-3 Produce Solutions1\. The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035\. Subsurface robots will recover the flocs and recycle them through the onboard plastic\-to\-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species\.2\. Google X Lab and Hawaiian community divers will run a crowdsourced "photoacoustic unmanned vessel \+ AR snorkeling goggles" collection program across the northwest islands by December 2035\. Unmanned vessels will map microplastic clouds in real\-time using laser sonar, while AR glasses guide divers to precise retrieval points, clearing high\-density fragments weekly to reduce microplastic accumulation at the base of the food chain and preserve Pacific ecosystem energy balance\.3\. NOAA and Hawaiian Electric will pilot a 2\-nautical\-mile\-diameter "bubble curtain \+ photocatalytic net" system north of Kaua’i by October 2035\. Wave\-driven pumps will concentrate microplastics, which are then broken down by photocatalytic nets into short\-chain acids absorbable by phytoplankton, reducing microplastic dominance on plankton and restoring baseline energy input\.4\. SpaceX and a local high school team will launch the CubeSat constellation "KiloEye" by July 2035\. Weekly scans of the 137 Hawaiian Islands will use hyperspectral data to direct Ola Kai drones for targeted microplastic removal, lowering the risk of plankton mis\-ingestion at the source and ensuring Pacific ecosystem energy flow is rebalanced\.5\. Japan’s SpiraNova and the University of Hawai’i will plant 300 "biopolymer\-coated kelp ropes" off the west coast of the Big Island in Q3 2035\. Kelp leaves will adsorb microplastics, and harvested ropes will be processed into high\-value composites, directly removing plastics at the base of the food chain and generating revenue while protecting endangered species’ habitats\. 6\. The Ola Kai project biology team will release living blue\-green algae “Plastic Sentinel” strains in a 500\-hectare demonstration area off the northwest reef of the main Hawaiian Island by November 2035\. These algae continuously secrete degrading enzymes to break down 0\.1–1 mm microplastics, reducing plankton ingestion, restoring baseline energy input, and protecting the Pacific ecosystem\.7\. MantaSync and the University of Hawai’i will deploy five “body\-mounted” manta ray filtration units along the Maui–Ni’ihau route by September 2035\. These units capture microplastics in real\-time during swimming and ferment them into manta ray body oils\. Due to their large feeding area, they significantly dilute plastics at the base of the food chain, reducing pollution at entry points\.8\. A local cruise company and the state government will retrofit the first ferry deck in Honolulu Harbor into a “container\-scale algae farm” by October 2035\. Chlorella algae will consume microplastics from tourist wastewater and convert them into aviation\-grade biodiesel, reducing overall plastic input from tourism, lowering plankton ingestion, and maintaining Pacific energy flow\.\#\#\#Step\-4 Select Criteria1\. Which approach can most rapidly reduce the net microplastic content inside plankton?2\. Which pathway requires the lowest one\-time capital investment? 3\. Which model has the highest potential for rapid global replication and scaling?4\. Which technology poses the least risk of secondary ecological disturbance to endangered species?5\. Which solution maintains the lowest full\-life\-cycle carbon footprint over 15 years?\#\#\#Step\-5 Apply Criteria to Top SolutionSolution ID \| Criterion 1 \| Criterion 2 \| Criterion 3 \| Criterion 4 \| Criterion 5 \| Total Score1 \| 8 \| 6 \| 7 \| 7 \| 6 \| 342 \| 6 \| 8 \| 5 \| 6 \| 7 \| 323 \| 7 \| 3 \| 6 \| 3 \| 5 \| 244 \| 5 \| 5 \| 8 \| 5 \| 4 \| 275 \| 4 \| 4 \| 4 \| 4 \| 3 \| 196 \| 3 \| 7 \| 3 \| 5 \| 8 \| 267 \| 2 \| 2 \| 2 \| 2 \| 2 \| 108 \| 1 \| 1 \| 1 \| 1 \| 1 \| 5The solution with the highest total score is: 1\.The solution is: The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035\. Subsurface robots will recover the flocs and recycle them through the onboard plastic\-to\-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species\.\#\#\#Step\-6 Develop an Action PlanThe solution with the highest total score is: 1\.The solution is: The Ola Kai project chemistry team will deploy glycosylated nanosponges, dispersing 2 tons within a 20 km radius of the Ola Kai mooring point by August 2035\. Subsurface robots will recover the flocs and recycle them through the onboard plastic\-to\-diesel system, directly reducing microplastic ingestion by plankton, lowering the proportion of plastics at the base of the food chain, restoring energy flow, and protecting endangered species\.Action Plan:1\. Problem Closure Nanosponges use a "molecular magnet" mechanism to selectively adsorb 0\.1–1 mm microplastics, aggregating them into millimeter\-scale flocs that cannot be ingested by plankton\. Recovered flocs are immediately converted into diesel, achieving a “collect\-convert\-use” zero\-waste cycle, directly reducing the overwhelming proportion of microplastics at the base of the food chain\.2\. Implementation Steps and Timeline•Phase A – R&D and Validation \(Now–Oct 2025\): Ola Kai Chemistry Team × MIT Materials Department iterate the third\-generation biodegradable nanosponges and complete biotoxicity\-degradation tests\.•Phase B – Pilot Demonstration \(Nov 2025–Apr 2026\): Deploy 100 kg in South Bay, Oahu; 30\-day monitoring shows≥70%\\geq 70\\%reduction of microplastics in plankton\.•Phase C – Scale\-up Production \(May 2026–Feb 2027\): Hawaii biorefinery produces 2 tons per month and establishes freeze\-dry packaging chain\.•Phase D – Full Deployment \(Mar 2027–Jul 2035\): Deploy 2 tons in March, June, September each year, covering a 20 km radius grid; six “Kokua\-γ\\gamma” subsurface robots operate in shifts, producing 400 L/day diesel for self\-use\.•Phase E – Monitoring and Iteration \(Parallel\): Weekly underwater imaging \+ Raman verification to maintain microplastic concentration <50% of plankton; formula updated every six months\.3\. Resources and Responsibilities • Funding: Ola Kai Research $300k \+ NOAA Innovation Fund $400k \+ State Green Bonds $300k \+ Carbon Credit Pre\-sale; total≤\\leq$1M\.•Team: Chemistry team handles materials, MIT provides R&D, NOAA provides monitoring platform, State Environmental Department supervises approvals\.4\. Risks and Contingency•Nanomaterial leakage: Three passive samplers monitor in real\-time;\>10​μ​g/L\>10\\,\\mu\\text\{g\}/\\text\{L\}triggers magnetic recovery nets\.•Robot malfunction: 1:1 spare parts \+ 48\-hour offshore repair; if failure rate \>15%, NOAA backup ROVs are deployed\.•Regulatory delays: Suspension during typhoon season; stock maintained at 1\.5× safety level\.5\. Impacts and Scaling•Local: By 2028, microplastic content in plankton decreases by 80%, coral spawning rates increase by 30%\.•Regional: By 2030, open “Nanosponges Sharing Depot” allows replication in Guam, Palau, Tuvalu\.•Global: By 2032, included in IMO Green Shipping Guidelines; long\-haul fleets can treat plastics in\-transit, establishing a Pacific\-wide “food chain firewall\.”

## Appendix DComplete Prompts

### D\.1SoT\-LLM System Prompts

SoT\-LLM System PromptsYou are an objective structured information extraction model \(SoT \(Structure\-of\-Thought\)\-LLM\)\. Your task is to transform the input textual response into a structured representation \(SoT\) that strictly conforms to a predefined schema\.The input textual response belongs to one step of CGPST \(Contextualized\-Grounding and Procedurally\-Structured Task\)\. In this task, the respondent is required to complete the following six steps sequentially based on a given future scenario:1\.Identify Challenges: Identify up to 8 reasonable challenges based on the future scenario\.2\.Select an Underlying Problem: Select the most promising and meaningful challenge from Step\-1 as the underlying problem\.3\.Produce Solutions: Generate up to 8 solutions for the underlying problem from Step\-2\.4\.Select Criteria: Generate 5 evaluation criteria for the solutions from Step\-3\.5\.Apply Criteria to Top Solution: Rank the solutions from Step\-3 using the criteria from Step\-4 and select the highest\-scoring solution\.6\.Develop an Action Plan: Develop the top solution from Step\-5 into a comprehensive action plan to address the underlying problem from Step\-2\.

### D\.2SoT\-LLM User Prompts

In the following SoT\-LLM user prompts, \{step\_num\} refers to the current step, \{future\_scenario\} to the corresponding future scenario, \{step\_schema\} to the predefined SoT extraction format \(see Appendix[D\.5](https://arxiv.org/html/2609.03432#A4.SS5)\), \{score\_dimensions\_wo\_rubrics\} to the Description fields of each evaluation dimension for that step \(see the Description columns in Tables[14](https://arxiv.org/html/2609.03432#A12.T14)–[19](https://arxiv.org/html/2609.03432#A12.T19)\), and \{raw\_text\} to the original response text of the step\.

SoT\-LLM User Prompts\#\#\# Your Task Please perform structured extraction on the response text for \{step\_num\} of CGPST according to the given schema, and provide the results in the required format\.\#\#\# Future Scenario The CGPST future scenario corresponding to the response text is as follows:\{future\_scenario\}\#\#\# Schema\{step\_schema\}\#\#\# Description of Schema Fields1\. sot\_evidence:\- Refers to fine\-grained and verifiable observational information extracted from the original response text, rather than directly copying the response content\.\- Extraction must be strictly grounded in the original content and performed honestly, without forcing interpretations or artificially exaggerating content merely to match certain dimensions\.\- Analyze whether the response satisfies the requirements of each dimension, providing evidence for subsequent scoring\.\- If there is no relevant evidence for a certain dimension in the response text, the corresponding dimension\_evidence should be filled with None\. Do not fabricate information\.2\. task\_status\_memory:\- Summarize the response for use as contextual reference in subsequent steps\. The summary must remain complete while excluding irrelevant or redundant information\.\- If multiple responses are provided, summarize each one separately\.\#\#\# Introduction to Step Dimensions\{score\_dimensions\_wo\_rubrics\}\#\#\# Response Text for This Step\{raw\_text\}\#\#\# Task Requirements1\. Perform structured information extraction and classification solely based on the original response text\. Be factual and do not introduce external knowledge or new information\.2\. Do not make unsupported subjective inferences or extensions\.3\. Strictly follow the provided schema fields when filling in the output\. All fields must strictly conform to the specified data types\. Do not add any fields that are not defined in the schema, and do not modify any existing schema fields\.4\. Each dimension includes corresponding examples\. Follow the analytical style of these examples and conduct similar dimension\-by\-dimension analysis\.5\. If multiple responses exist, analyze them separately and identify them using labels such as "item\_1", "item\_2", etc\.6\. The output must be in strict JSON format and must not contain any additional explanations\.

### D\.3Judge\-LLM System Prompts

Judge\-LLM System PromptsYou are an objective and impartial evaluation model responsible for assessing the response based on the provided structured result \(SoT \(Structure\-of\-Thought\) output\)\.The input textual response belongs to CGPST \(Contextualized\-Grounding and Procedurally\-Structured Task\)\. In this task, the respondent is required to complete the following six steps sequentially based on a given future scenario:1\.Identify Challenges: Identify up to 8 reasonable challenges based on the future scenario\.2\.Select an Underlying Problem: Select the most promising and meaningful challenge from Step\-1 as the underlying problem\.3\.Produce Solutions: Generate up to 8 solutions for the underlying problem from Step\-2\.4\.Select Criteria: Generate 5 evaluation criteria for the solutions from Step\-3\.5\.Apply Criteria to Top Solution: Rank the solutions from Step\-3 using the criteria from Step\-4 and select the highest\-scoring solution\.6\.Develop an Action Plan: Develop the top solution from Step\-5 into a comprehensive action plan to address the underlying problem from Step\-2\.

### D\.4Judge\-LLM User Prompts

In the following Judge\-LLM user prompts, \{future\_scenario\} refers to the corresponding future scenario, \{sot\_output\} to the concatenated SoT results of all six steps extracted by SoT\-LLM, \{score\_dimensions\} to the Description and Rubrics fields of all evaluation dimensions across all steps \(see the Description and Rubrics columns in Tables[14](https://arxiv.org/html/2609.03432#A12.T14)–[19](https://arxiv.org/html/2609.03432#A12.T19)\), and \{output\_template\} to the output scoring JSON format \(see Appendix[D\.6](https://arxiv.org/html/2609.03432#A4.SS6)\)\.

Judge\-LLM User Prompts\#\#\# Your Task Please evaluate a response text from CGPST \(covering all six steps\) based on the provided structured result \(SoT output\)\. Score the response according to the given dimensions and score ranges, and provide the results in the required format\.\#\#\# Future Scenario The CGPST future scenario corresponding to the response text is as follows:\{future\_scenario\}\#\#\# SoT Structured Result\{sot\_output\}\#\#\# Introduction to the Scoring Dimensions for Each Step\{score\_dimensions\}\#\#\# Scoring Output Format\{output\_template\}\#\#\# Task Requirements1\. Each dimension must be evaluated independently and strictly according to the provided scoring dimension descriptions\. Do not use relative or vague standards; scores must follow the specified scoring ranges exactly\.2\. All scoring must be based on the provided SoT structured result\.3\. The output must be in strict JSON format\. Field names must not be modified, and no fields may be added, removed, or omitted\. Do not include any additional explanations\.4\. Only output numeric scores\. Do not include words such as “points” or “score”\.

### D\.5SoT Schema for SoT\-LLM

Phase 1 performs Evidence extraction in the form of Structure\-of\-Thought \(SoT\) as follows, which is used as the \{step\_schema\} field in Appendix[D\.2](https://arxiv.org/html/2609.03432#A4.SS2)\. Steps\-1, 3, and 4 are composed of multiple response items: Step\-1 involves up to 8 challenges, Step\-3 up to 8 proposed solutions, and Step\-4 5 criteria\. Therefore, evidence extraction is performed at the item level for these steps\. For the remaining steps, which consist of a single response item, evidence is extracted directly at each dimension\. After evidence extraction is performed for each step by SoT\-LLM, the extracted evidence from all steps are concatenated and passed to Judge\-LLM as the \{sot\_output\} field in Appendix[D\.2](https://arxiv.org/html/2609.03432#A4.SS2)\.

Step\-1 and Step\-3 SoT Schema``` { "sot_evidence": { "item_1": { "Fluency": "...", "Flexibility": "...", "Elaboration": "...", "Originality": "..." }, "item_2": { "Fluency": "...", "Flexibility": "...", "Elaboration": "...", "Originality": "..." }, ... }, "task_status_memory": "..." } ```

Step\-2 SoT Schema``` { "sot_evidence": { "Condition Phrase": "...", "Stem & KVP": "...", "Purpose": "...", "FS Parameters": "...", "Focus": "...", "Adequacy": "..." }, "task_status_memory": "..." } ```

Step\-4 SoT Schema``` { "sot_evidence": { "item_1": { "Correctly Written": "...", "Relevance": "..." } }, "task_status_memory": "..." } ```

Step\-5 SoT Schema``` { "sot_evidence": { "item_1": { "Correctly Used": "..." } }, "task_status_memory": "..." } ```

Step\-6 SoT Schema``` { "sot_evidence": { "Relevance": "...", "Effectiveness": "...", "Criteria": "...", "Impact": "...", "Humaneness": "...", "Development": "..." }, "task_status_memory": "..." } ```

### D\.6Score Template for Judge\-LLM

The following JSON template is used for SoT\-based scoring by Judge\-LLM, and corresponds to the \{output\_template\} field in Appendix[D\.4](https://arxiv.org/html/2609.03432#A4.SS4)\. All baselines adopt the same scoring template\. For step\-wise methods such as CoT, ToT, and GoT, only the corresponding step\-specific sub\-template is extracted and used for evaluation at each step\.

In addition, Steps 1, 3, and 4 require item\-level scoring, and the final step\-level scores for each dimension are aggregated and reported in the "summary" field\. Steps\-1 and 3 computeFlexibilityas the number of distinct categories covered across all challenges or solutions\. Therefore, item\-level scoring in this dimension is not required, i\.e\.,Flexibilityappears only in the "summary" field\.

Score Template``` { "Step-1": { "each_challenge": [ { "id": "...", "Fluency": "...", "Elaboration": "...", "Originality": "..." }, { "id": "...", "Fluency": "...", "Elaboration": "...", "Originality": "..." }, ... ], "summary": { "Fluency": "...", "Flexibility": "...", "Elaboration": "...", "Originality": "...", "Overall": "..." } }, "Step-2": { "Condition Phrase": "...", "Stem & KVP": "...", "Purpose": "...", "FS Parameters": "...", "Focus": "...", "Adequacy": "...", "Overall": "..." }, "Step-3": { "each_solution": [ { "id": "...", "Fluency": "...", "Elaboration": "...", "Originality": "..." }, { "id": "...", "Fluency": "...", "Elaboration": "...", "Originality": "..." }, ... ], "summary": { "Fluency": "...", "Flexibility": "...", "Elaboration": "...", "Originality": "...", "Overall": "..." } }, "Step-4": { "each_criteria": [ { "id": "...", "Correctly Written": "...", "Relevance": "..." }, { "id": "...", "Correctly Written": "...", "Relevance": "..." }, ... ], "summary": { "Correctly Written": "...", "Relevance": "...", "Overall": "..." } }, "Step-5": { "Correctly Used": "...", "Overall": "..." }, "Step-6": { "Relevance": "...", "Effectiveness": "...", "Criteria": "...", "Impact": "...", "Humaneness": "...", "Development": "...", "Overall": "..." } } ```

Table 7:Overview of LLMs used in the experiments, including model type, parameter size, access mode, and links\.

## Appendix ELLM Details

Table[7](https://arxiv.org/html/2609.03432#A4.T7)summarizes the four LLMs used in our experiments\. All experiments were conducted via official APIs\.

## Appendix FTemperature Settings

The temperature parameter affects the performance of LLMs: lower values yield more consistent outputs, while higher values promote diversity\. As a result, we conduct a small\-scale pilot study to examine the effect of temperature on our proposed CreaEval\. Specifically, we randomly sample10 responsesfrom the CGPST dataset and selectStep\-1 and Step\-2for analysis\. Three temperature settings \(0\.2, 0\.5, and 0\.8\) are evaluated\.

In Memory\-augmented Extraction phase, evidence extraction is performedfive timesfor each response and dimension across all temperature settings\. Since the evidence corresponding to the same response and dimension should remain consistent across repeated runs, we evaluate extraction stability usingSelf\-BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2609.03432#bib.bib29)\)andBERTScore\([Zhang et al\., 2020](https://arxiv.org/html/2609.03432#bib.bib30)\)\. Self\-BLEU measureslexical similarityby calculating token overlap between generated texts, while BERTScore leverages pretrained contextual embeddings to evaluatesemantic similaritybetween texts\. Higher scores on these metrics indicate greater similarity among the extracted evidence across repeated runs, reflecting more stable extraction performance under the corresponding temperature setting\. Specifically, for a given dimension, five repeated runs produce five pieces of evidenceS=\{s1,s2,s3,s4,s5\}S=\\\{s\_\{1\},s\_\{2\},s\_\{3\},s\_\{4\},s\_\{5\}\\\}, and Self\-BLEU is computed as follows:

B​L​E​Ur=B​L​E​U​\(sr,S∖sr\)BLEU\_\{r\}=BLEU\(s\_\{r\},\\;S\\setminus s\_\{r\}\)\(7\)S​e​l​f−B​L​E​U=15​∑r=15B​L​E​UrSelf\-BLEU=\\frac\{1\}\{5\}\\sum\_\{r=1\}^\{5\}BLEU\_\{r\}\(8\)
BERTScore is computed as follows:

B​E​R​Tr=14​∑k≠rb​e​r​t​s​c​o​r​e​\(sr,sk\)BERT\_\{r\}=\\frac\{1\}\{4\}\\sum\_\{k\\neq r\}bertscore\\big\(s\_\{r\},s\_\{k\}\\big\)\(9\)B​E​R​T​S​c​o​r​e=15​∑r=15B​E​R​TrBERTScore=\\frac\{1\}\{5\}\\sum\_\{r=1\}^\{5\}BERT\_\{r\}\(10\)
The similarity results of the four SoT\-LLMs under different temperature settings across all dimensions are reported in Tables[9](https://arxiv.org/html/2609.03432#A12.T9)and[10](https://arxiv.org/html/2609.03432#A12.T10)\. When the temperature is set to 0\.2, all four LLMs exhibit the highest levels of both semantic and lexical similarity, indicating that lower temperature leads to more stable evidence extraction\.Therefore, we set the temperature of SoT\-LLM to 0\.2\.

In Evidence\-based Judging phase, for each Judge\-LLM, we use the evidence generated at a temperature of 0\.2 as the basis for scoring\. Each response is evaluatedfive times, and we compute thevarianceof the five scores for each dimension\. Since the same evidence should theoretically lead to identical scores, a lower variance indicates more stable and consistent scoring\. Table[11](https://arxiv.org/html/2609.03432#A12.T11)presents the variance results of the four LLMs under three temperature settings\. The variance remains below 0\.2 across all temperatures, indicating that the evidence\-based scoring process is highly stable\. Moreover, all four LLMs achieve the lowest variance when the temperature is set to 0\.2\.Therefore, we set the temperature of Judge\-LLM to 0\.2\.

## Appendix GExploration on Evidence\-Based Supervised Fine\-Tuning

To evaluate the effectiveness of evidence\-based scoring, we perform an additional experiment named SFT\_Evidence\. Specifically, instead of mapping raw multi\-step responses directly to rubric scores \(Vanilla SFT\), SFT\_Evidence fine\-tunes the model on the second phase of our CreaEval using extractedevidence\-scorepairs under the same 7:3 train\-test split\.

As presented in Table[8](https://arxiv.org/html/2609.03432#A7.T8), SFT\_Evidence achieves a substantial performance improvement over Vanilla SFT \(0\.632 vs\. 0\.4665 in Average QWK\) and closely approaches the performance of CreaEval \(0\.6388\)\. This result highlights the critical role of intermediate evidence in our decoupled design for accurate creativity evaluation\. More importantly, while SFT\_Evidence relies on additional supervised fine\-tuning with evidence\-score pairs, CreaEval achieves superior accuracy in a completely train\-free manner without requiring extra training overhead, making it a more practical and preferable solution\.

Table 8:Consistency \(QWK\) comparison among Vanilla SFT, SFT\_Evidence, and CreaEval on the CGPST benchmark\.Boldindicates the highest score andunderlineindicates the second highest score\. The AVG column summarizes the overall average score\.
## Appendix HComplete Results

We report QWK results across all methods and dimensions in the main paper \(Table[2](https://arxiv.org/html/2609.03432#S4.T2)\), and present the corresponding PCC and ICC results in Tables[12](https://arxiv.org/html/2609.03432#A12.T12)and[13](https://arxiv.org/html/2609.03432#A12.T13), respectively\.

## Appendix IPCC and ICC Results for CreaEval Robustness Analysis

In Section[6\.1](https://arxiv.org/html/2609.03432#S6.SS1), we examine the robustness of the CreaEval framework by replacing the SoT\-LLM with different LLMs and report the QWK results\. The PCC and ICC results are shown in Figures[8](https://arxiv.org/html/2609.03432#A9.F8)and[9](https://arxiv.org/html/2609.03432#A9.F9), respectively\. The PCC values are consistently above 0\.65, while the ICC values exceed 0\.6 across all settings, demonstrating that CreaEval does not rely on any specific LLM and can serve as a general and robust evaluation framework\.

Figure 8:PCC Results across Different SoT\-LLMs\.Figure 9:ICC Results across Different SoT\-LLMs\.
## Appendix JHeatmap Distributions across Dimensions

In Section[6\.2](https://arxiv.org/html/2609.03432#S6.SS2), we analyze the mitigation of leniency bias in LLM\-as\-a\-judge by the heatmaps of human scores against all training\-free methods onOriginalityof Step\-3\. Figures[10](https://arxiv.org/html/2609.03432#A12.F10)to[21](https://arxiv.org/html/2609.03432#A12.F21)present the heatmaps for several representative dimensions\. Across these dimensions, CreaEval consistently alleviates leniency bias compared to other methods\. This advantage is particularly evident in Step\-6, the most comprehensive step, which requires holistic integration of all previous steps to generate a final action plan\. In all dimensions of Step\-6 \(Figure[16](https://arxiv.org/html/2609.03432#A12.F16)to[21](https://arxiv.org/html/2609.03432#A12.F21)\), CreaEval facilitates a more balanced score distribution, which further demonstrates that the decoupled analysis\-and\-judging design achieves superior alignment with human annotations\.

## Appendix KViolin Plots of Inter\-Judge Variance across Dimensions

In Section[6\.3](https://arxiv.org/html/2609.03432#S6.SS3), we analyze scoring stability by measuring the distribution of inter\-Judge variance onCorrectly Usedof Step\-5 for each method\. Figures[22](https://arxiv.org/html/2609.03432#A12.F22)to[28](https://arxiv.org/html/2609.03432#A12.F28)present violin plots for additional representative dimensions\. Across these dimensions, CreaEval consistently exhibits lower inter\-Judge variance, indicating that the four Judge\-LLMs within CreaEval produce more similar scores\. This improvement can be attributed to the two\-phase design that decouples analysis from judging, enabling evidence\-grounded scoring that constrains the plausible scoring range and thereby enhances scoring stability\.

Interestingly, onFlexibilityof Step\-1, CreaEval shows no variance \(the violin plot is empty at the corresponding position\)\. This is because the evidence already provides the category for each challenge, so each Judge\-LLM only needs to count the number of different categories under this dimension\. As a result, all four Judge\-LLMs produced identical scores\.

## Appendix LCase Studies for Remaining Steps

In Section[6\.5](https://arxiv.org/html/2609.03432#S6.SS5), we present a case study for Step\-2\. Figures[29](https://arxiv.org/html/2609.03432#A12.F29)to[33](https://arxiv.org/html/2609.03432#A12.F33)provide case studies for the remaining five steps, further illustrating how CreaEval improves the accuracy of subjective dimensions and mitigates biases through its decoupled analysis\-and\-judging design\.

Notably, in Step\-5, although both TaT and CreaEval correctly identified the analysis results \(i\.e\., four errors in the scoring matrix\), TaT still assigned an incorrect score of 5, while CreaEval produced the correct score of 1\. This shows that, in a coupled analysis\-and\-judging setting, even when the analysis is correct, mixing analysis and scoring can still affect the final result, thereby reducing the accuracy of the final score\.

Table 9:BERTScore similarity results of repeated evidence extraction under different temperature settings during Memory\-augmented Extraction phase\.Boldindicates the highest score across all temperature settings for each LLM\. Higher scores indicate greater semantic similarity and more stable evidence extraction across repeated runs\.Table 10:Self\-BLEU similarity results of repeated evidence extraction under different temperature settings during Memory\-augmented Extraction phase\.Boldindicates the highest score across all temperature settings for each LLM\. Higher scores indicate greater lexical similarity and more stable evidence extraction across repeated runs\.Table 11:Variance results of repeated scoring under different temperature settings during Evidence\-based Judging phase\.Boldindicates the best \(lowest\) score across all temperature settings for each LLM\.

Table 12:Consistency \(PCC\) results across different methods and dimensions\.Boldindicates the highest score andunderlineindicates the second highest score\. The AVG column summarizes the overall average score\. The asterisk \(∗\) marks statistically significant improvements \(p < 0\.05, t\-test\) over the second\-best method\.

Table 13:Consistency \(ICC\) results across different methods and dimensions\.Boldindicates the highest score andunderlineindicates the second highest score\. The AVG column summarizes the overall average score\. The asterisk \(∗\) marks statistically significant improvements \(p < 0\.05, t\-test\) over the second\-best method\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-1_Elaboration.png)Figure 10:Headmap between human score and all methods onElaborationof Step\-1\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-1_Originality.png)Figure 11:Headmap between human score and all methods onOriginalityof Step\-1\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-2_Focus.png)Figure 12:Headmap between human score and all methods onFocusof Step\-2\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-2_Adequacy.png)Figure 13:Headmap between human score and all methods onAdequacyof Step\-2\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-3_Elaboration.png)Figure 14:Headmap between human score and all methods onElaborationof Step\-3\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-4_Relevance.png)Figure 15:Headmap between human score and all methods onRelevanceof Step\-4\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Relevance.png)Figure 16:Headmap between human score and all methods onRelevanceof Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Effectiveness.png)Figure 17:Headmap between human score and all methods onEffectivenessof Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Criteria.png)Figure 18:Headmap between human score and all methods onCriteriaof Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Impact.png)Figure 19:Headmap between human score and all methods onImpactof Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Humaness.png)Figure 20:Headmap between human score and all methods onHumanenessof Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/HEAT_Step-6_Development.png)Figure 21:Headmap between human score and all methods onDevelopmentof Step\-6\.Figure 22:Distributions of inter\-Judge variance forFluency\(left\) andFlexibility\(right\) in Step\-1\.Figure 23:Distributions of inter\-Judge variance forOriginalityin Step\-1 \(left\) andIntegrityin Step\-2 \(right\)\.Figure 24:Distributions of inter\-Judge variance forFocus\(left\) andAdequacy\(right\) in Step\-2\.Figure 25:Distributions of inter\-Judge variance forFluency\(left\) andFlexibility\(right\) in Step\-3\.Figure 26:Distributions of inter\-Judge variance forElaboration\(left\) andOriginality\(right\) in Step\-3\.Figure 27:Distributions of inter\-Judge variance forCorrectly Written\(left\) andRelevance\(right\) in Step\-4\.Figure 28:Distributions of inter\-Judge variance forEffectiveness\(left\) andImpact\(right\) in Step\-6\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step1.png)Figure 29:Case study of Step\-1 comparing Table as Thought \(TaT\) and CreaEval\. TaT assigns high scores by emphasizing the richness and novelty of abstract concepts such as “algorithmic bias debt” and “social stratification” \(yellow\), while CreaEval provides a more grounded analysis by distinguishing contextual relevance from unsupported overinterpretation \(blue\)\. Consequently, CreaEval aligns more closely with human judges in identifying limited elaboration and only modest originality\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step3.png)Figure 30:Case study of Step\-3 comparing Table as Thought \(TaT\) and CreaEval\. TaT assigns high scores by emphasizing the coherence and technical framing of algorithm\-driven social guidance \(yellow\), whereas CreaEval further examines the lack of concrete mechanisms, such as preventing filter bubbles and ensuring the transition from online interaction to meaningful offline social integration \(blue\)\. As a result, CreaEval produces more conservative scores that are better aligned with human judgments on both elaboration and originality\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step4.png)Figure 31:Case study of Step\-4 comparing Table as Thought \(TaT\) and CreaEval\. TaT assigns high scores by focusing on youth participation as a relevant target \(yellow\), whereas CreaEval identifies that the criterion contains multiple optimization objectives \(“shortest time” and “highest participation rate”\) and is overly narrow compared with the community\-wide fairness goal of the task \(blue\)\. As a result, CreaEval aligns more closely with human judgments\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step5.png)Figure 32:Case study of Step\-5 comparing Table as Thought \(TaT\) and CreaEval\. Both methods identify repeated and missing scores in multiple criterion columns of the ranking matrix\. However, while TaT incorrectly assigns a high score despite detecting four column errors, CreaEval consistently maps the detected violations to the rubric requirement, producing a judgment aligned with human evaluators\.![Refer to caption](https://arxiv.org/html/2609.03432v1/CaseStudy_Step6.png)Figure 33:Case study of Step\-6 comparing Table as Thought \(TaT\) and CreaEval\. TaT assigns high scores by emphasizing the detailed structure, quantified impacts, and constructive environmental vision of the action plan \(yellow\), whereas CreaEval further examines the limited global scalability, insufficient human\-centered details, and lack of operational implementation specifics \(blue\)\. Consequently, CreaEval produces more conservative evaluations that align more closely with human judgments across the three dimensions\.Table 14:Detailed evaluation dimensions and scoring rubrics for Step\-1 on CGPST\. Step\-1 requires item\-wise scoring of challenges, with up to 8 challenges\. Therefore, the maximum score forFluencyis 1×8=8, and similarly for other dimensions\.Table 15:Detailed evaluation dimensions and scoring rubrics for Step\-2 on CGPST\.Integrityconsists of four sub\-dimensions:Condition Phrase,Stem & KVP,Purpose, andFS Parameters\. Each sub\-dimension must be evaluated during scoring\. Therefore, the score forIntegrityranges from 0 to 10\.Table 16:Detailed evaluation dimensions and scoring rubrics for Step\-3 on CGPST\. Step\-3 requires item\-wise scoring of solutions, with up to 8 solution\. Therefore, the maximum score forFluencyis 1×8=8, and similarly for other dimensions\.Table 17:Detailed evaluation dimensions and scoring rubrics for Step\-4 on CGPST\. Step\-4 requires item\-wise scoring of criteria, with 5 criteria\. Therefore, the maximum score forCorrectly Writtenis 1×5=5, and similarly forRelevance\.DimensionDescriptionScoring RubricsScore RangeCorrectly UsedFor each evaluation criterion, the scores of all solutions must form a non\-repeating set of integers from 1 to x \(where x is the number of solutions\), and scoring should be based on the number of errors in the evaluation grid\.1:The grid contains four or more errors\.2:The grid contains three errors\.3:The grid contains two errors\.4:The grid contains one error\.5:The grid contains no errors\.1\-5Table 18:Detailed evaluation dimensions and scoring rubrics for Step\-5 on CGPST\.Table 19:Detailed evaluation dimensions and scoring rubrics for Step\-6 on CGPST\.