A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs

# A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs
Source: [https://arxiv.org/html/2608.26177](https://arxiv.org/html/2608.26177)
###### Abstract

Long\-form generation exposes fundamental limitations of large language models\. Even 70B\-parameter models exhibit length collapse at 16k\-token outputs, and multi\-chapter stories frequently trigger the attribute drift characteristic of the “lost\-in\-the\-middle” effect\. The “outline\-first, write\-later” paradigm has gained wide adoption, yet existing research evaluates the final writing rather than the outline itself, conflating two evaluation objects that should be decoupled\. We construct a unified head\-to\-head benchmark covering 7 representative long\-form generation frameworks across 3 generation granularities — single\-chapter, multi\-chapter, and whole\-book — and propose an anchor\-based LLM\-as\-a\-judge protocol that directly assesses outlines against the source text on a 5\-point anchored scale\. Across 21 framework\-granularity cells,no single framework dominates; performance depends on the match between a framework’s intrinsic output form and the target granularity\.SuperWriter ranks first in the length\-constrained single\-chapter mode, but this advantage degrades in whole\-book mode\.The outline\-side ranking correlates only moderately with the writing\-side ranking, supporting the outline–writing decoupling principle\. Compute constraints limit the writing\-side evaluation to a subset of cases; follow\-up experiments will expand the sample size and add cross\-model evaluators to enable stronger statistical inference\.

Keywords:long\-form generation, large language models, outline\-driven generation frameworks, controlled comparison, length compliance, long\-range coherence, LLM\-as\-a\-judge

## 1Introduction

Recent progress in long\-form generation has fueled strong interest in outline\-driven frameworks, yet how to evaluate the outline stage independently of the final text remains unclear\. As task length scales from a few hundred characters to several thousand, three intertwined constraints come to define the scope of the problem: structural coherence, long\-range consistency, and length compliance\. Among these, long\-range consistency is the most prominent — the Transformer’s attention mechanism suffers from the well\-known “lost\-in\-the\-middle” effect\[[10](https://arxiv.org/html/2608.26177#bib.bib6)\], where information in the middle of the context receives systematically lower weight, leading to character drift and plot contradictions\. Length compliance poses an independent bottleneck:LongGenBench\[[20](https://arxiv.org/html/2608.26177#bib.bib7)\]shows that even 70B\-parameter models reach only11\.5%11\.5\\%on STIC\-2 at 16k/32k\-token outputs, andHelloBench\[[13](https://arxiv.org/html/2608.26177#bib.bib1)\]finds that models often stop near2,0002\{,\}000characters when asked to generate4,0004\{,\}000\.These failure modes motivate our focus on the outline stage\.Prior work — LongWriter, Re3, DOC, and StoryWriter — repeatedly identifies outlines as central to long\-range coherence and builds outline planning as the core mechanism\. Multi\-agent reflective revision reaches the same conclusion\.

Existing research addresses long\-form generation through three intertwined paradigms \(see Section[2](https://arxiv.org/html/2608.26177#S2)for a survey\)\.The “plan\-first, write\-later” paradigmseparates outline generation from draft composition\. It originates in early neural story generation\[[4](https://arxiv.org/html/2608.26177#bib.bib8)\]and extends to recursive reprompting[24](https://arxiv.org/html/2608.26177#bib.bib4), fine\-grained outline control[23](https://arxiv.org/html/2608.26177#bib.bib23), and natural\-language memory simulation[27](https://arxiv.org/html/2608.26177#bib.bib10)\.The multi\-agent collaboration paradigmtreats “conversational agents \+ procedural dialogue flow” as a general pattern for complex LLM tasks[19](https://arxiv.org/html/2608.26177#bib.bib26),[7](https://arxiv.org/html/2608.26177#bib.bib12): it encodes standard operating procedures as prompt sequences, coordinates planning and writing agents, or iteratively refines outlines through multi\-turn dialogue\.The reflection\-and\-revision paradigmvalidates that the “evaluate–reflect–revise” loop plays a critical role in long\-form coherence, distilling self\-criticism into episodic memory through natural\-language reflection[17](https://arxiv.org/html/2608.26177#bib.bib14),[11](https://arxiv.org/html/2608.26177#bib.bib19)\.

Evaluation methodology has evolved in parallel: from a BLEU/ROUGE\-dominated landscape, to multi\-dimensional fine\-grained scoring as represented by HANNA\[[3](https://arxiv.org/html/2608.26177#bib.bib3)\]and WriteJudge\[[21](https://arxiv.org/html/2608.26177#bib.bib20)\]\. The LLM\-as\-a\-judge paradigm has been validated at over80%80\\%agreement with humans\[[26](https://arxiv.org/html/2608.26177#bib.bib16),[6](https://arxiv.org/html/2608.26177#bib.bib17)\], yet position bias, verbosity bias, and self\-preference bias persist\. For long\-form evaluation, ExPerT\[[15](https://arxiv.org/html/2608.26177#bib.bib18)\], ProxyQA\[[18](https://arxiv.org/html/2608.26177#bib.bib21)\], and WritingBench\[[22](https://arxiv.org/html/2608.26177#bib.bib2)\]reduce dependence on gold\-standard reference answers through atomic aspect extraction, proxy questions, and query\-relevant scoring rubrics\.However, a critical evaluation gap remains: existing writing\-side evaluation — that is, scores from HANNA and WriteJudge on writing outputs — evaluates the writing rather than the outline itself\.When researchers ask “which framework produces better outlines?”, they can only infer backward from writing\-side metrics, conflating outline\-side quality with writing\-side quality and obscuring genuine differences in outline design across frameworks\. The insight behind our dual\-track design is that the outline and the writing are two decoupled evaluation objects that share only the source text\.

This gap motivates the present work\. We construct a unified controlled benchmark and propose a dual\-track evaluation protocol\.Our contributions are threefold:

#### \(1\) A unified head\-to\-head benchmark covering 7 long\-form generation frameworks across 3 generation granularities\.

We compare representative plan\-first\-write\-later frameworks and naive baselines on a unified LLM backend, with faithfulness to the original\-paper skeletons controlled across 21 framework\-granularity cells\.

#### \(2\) An anchor\-based LLM\-as\-a\-judge protocol for direct outline evaluation\.

The protocol anchors the rating scale to the source text as a 5\-point anchor, decoupling outline evaluation from writing evaluation\. The two complementary tracks — approximately 89 cases on the outline side and a subset of samples on the writing side — share only the source text\.

#### \(3\) Four empirical heuristics and one decoupling principle

\(first reported in Section[4\.5](https://arxiv.org/html/2608.26177#S4.SS5)\)\. No single framework dominates across all granularities; the outline\-side and writing\-side rankings correlate only moderately, supporting the outline–writing decoupling principle\.

Compute constraints limit the writing\-side evaluation to a subset of cases; follow\-up experiments will expand the sample size and add additional models as evaluators for controlled comparisons\.

## 2Related Work

Long\-form generation has converged on a shared organizing principle: explicit pre\-writing planning, multi\-agent collaboration, and post\-hoc reflection constitute the three dominant paradigms for controlling long\-range coherence\. We review each in turn and situate our benchmark against them\.

#### Outline\-driven long\-form generation\.

“Outline\-first, write\-later” is a classic pattern of human writing and the first paradigm studied systematically in long\-form generation\. Fanet al\.\[[4](https://arxiv.org/html/2608.26177#bib.bib8)\]introduced it into neural story generation; Yaoet al\.\[[25](https://arxiv.org/html/2608.26177#bib.bib22)\]proposed Plan\-and\-Write, explicitly splitting generation into planning and writing phases; Goldfarb\-Tarrantet al\.\[[5](https://arxiv.org/html/2608.26177#bib.bib9)\]demonstrated through user studies that plan\-write\-revise yields10%10\\%–50%50\\%quality gains\. In the era of LLMs, this paradigm has extended further: Yanget al\.’s Re3\[[24](https://arxiv.org/html/2608.26177#bib.bib4)\]and DOC\[[23](https://arxiv.org/html/2608.26177#bib.bib23)\]improve long\-range coherence through recursive reprompting and fine\-grained outline control; Zhouet al\.\[[27](https://arxiv.org/html/2608.26177#bib.bib10)\]simulates LSTM short/long\-term memory with natural\-language components\. The plan\-first\-write\-later frameworks compared here belong to this lineage, with distinct design orientations — cognitive writing, dynamic hierarchical outlines, event graphs, and hard length constraints — that unify outline generation and expansion\.Unlike work that proposes a single new framework, we construct a head\-to\-head benchmark spanning multiple plan\-first\-write\-later frameworks\.

#### Multi\-agent and reflection\-based revision\.

A parallel line of research models long\-form generation as multi\-agent collaboration or process supervision\. Wuet al\.\[[19](https://arxiv.org/html/2608.26177#bib.bib26)\]established “conversational agents \+ procedural dialogue flow” as a general pattern for complex LLM tasks; Honget al\.\[[7](https://arxiv.org/html/2608.26177#bib.bib12)\]encoded standard operating procedures as prompt sequences coordinated by multi\-role agents through a shared message pool; Huotet al\.\[[8](https://arxiv.org/html/2608.26177#bib.bib11)\]and Shaoet al\.\[[16](https://arxiv.org/html/2608.26177#bib.bib13)\]iteratively refine outlines through planning/writing agent coordination or multi\-turn dialogue\. On the reflection and process\-supervision side, Reflexion\[[17](https://arxiv.org/html/2608.26177#bib.bib14)\]distills self\-criticism into episodic memory through natural\-language reflection; DPO\[[14](https://arxiv.org/html/2608.26177#bib.bib24)\]converts preference alignment into a binary cross\-entropy loss; LongDPO\[[12](https://arxiv.org/html/2608.26177#bib.bib25)\]combines stepwise DPO with a global memory pool to optimize long\-form generation specifically\.Unlike these works that propose a single framework, we provide a controlled comparison platform on which they can be evaluated side\-by\-side\.

#### Evaluation methodology and direct outline evaluation\.

Evaluation protocols have shifted from BLEU/ROUGE dominance to multi\-dimensional fine\-grained scoring\. HANNA\[[3](https://arxiv.org/html/2608.26177#bib.bib3)\]and WriteJudge\[[21](https://arxiv.org/html/2608.26177#bib.bib20)\]assess the quality of generated stories along multiple dimensions; the LLM\-as\-a\-judge paradigm has reached85%85\\%non\-tie agreement with humans on MT\-Bench\[[26](https://arxiv.org/html/2608.26177#bib.bib16)\], yet position bias, verbosity bias, and self\-preference bias persist; ExPerT\[[15](https://arxiv.org/html/2608.26177#bib.bib18)\], ProxyQA\[[18](https://arxiv.org/html/2608.26177#bib.bib21)\], and WritingBench\[[22](https://arxiv.org/html/2608.26177#bib.bib2)\]respectively reduce dependence on gold\-standard reference answers through atomic aspect extraction, proxy questions, and query\-relevant scoring rubrics\.However, existing writing\-side evaluation targets the writing output rather than the outline itself — this is the methodological gap that our anchor\-based LLM\-as\-a\-judge protocol fills\.We integrate the HANNA scoring, the WebNovelBench narrative\-quality framework\[[9](https://arxiv.org/html/2608.26177#bib.bib15)\], the LongBench\-Write length score\[[2](https://arxiv.org/html/2608.26177#bib.bib5)\], and WritingBench’s query\-relevant scoring rubric into a unified protocol covering 7 frameworks across 3 granularities, decoupling the outline stage from the expansion stage into an outline\-side primary track plus a writing\-side auxiliary track\.

## 3Method

We do not propose a new generation framework\. Instead, we conduct a head\-to\-head comparison of existing long\-form generation frameworks on a unified benchmark\. This section provides the formal task description, experimental design, and evaluation protocol\.

### 3\.1Task and Definitions

A long\-form generation framework is defined asℱ=⟨P,𝒪,𝒲⟩\\mathcal\{F\}=\\langle P,\\,\\mathcal\{O\},\\,\\mathcal\{W\}\\rangle, wherePPis the prompt orchestration across stages,𝒪\\mathcal\{O\}is the intermediate representation \(natural\-language paragraphs, structured JSON, narrative outline prose, or none\), and𝒲\\mathcal\{W\}is the writing strategy that maps the final outline to the long output\. Given a source textxxand a target lengthLL, the framework outputs a long textyy\. Our comparison variesℱ\\mathcal\{F\}across 7 representative frameworks while holding the LLM backend, evaluator model, and metric family fixed\. We invert the original “premise→\\rightarrowlong text” forward generation setting into a “long text→\\rightarrowoutline” backward extraction setting \(hereafterTask Inversion\) — all 7 frameworks face existing text rather than user instructions\. This inversion introduces three systematic side effects: \(i\) Dome’s character slots are empty on the execution path; \(ii\) StoryWriter’s chapter\-count range must be adjusted by mode; \(iii\) the evaluation criterion shifts from “generation quality” to “outline alignment\.”

#### Controlled variables\.

We fix the following: \(1\) Unified writing backend — all 7 frameworks share the same LLM \(deepseek\-v4\-flash\); \(2\) Unified evaluator — the LLM evaluator is the same model as the generator; self\-preference bias is not mitigated in this study; \(3\) Unified data interface and contract — input semantics align across the 3 loading modes, and all outputs are produced under unified JSON fields; \(4\) Unified metric family, with dimensions activated per mode — D1\_cross\_chapter\_consistency activates only in whole\-book mode, M1\_continuity\_with\_prior only in multi\-chapter mode, and M7\_CR only in multi\-chapter / whole\-book modes; \(5\) Skeleton alignment — the prompt orchestration order, number of stages, and key slot constraints of all 7 frameworks are isomorphic to those in the original papers or official code\.

#### Uncontrolled differences\.

This experiment is a controlled comparison, not a fully univariate design: \(i\) Sampling\-parameter differences — each framework uses its original paper’s default temperature \(CogWriter0\.10\.1, SuperWriter0\.60\.6, StoryWriter stage\-11\.01\.0/ stage\-20\.50\.5, others0\.70\.7\), not a unified value; \(ii\) Prompt\-level bias — few\-shot deletion, inlined\-JSON comment replacement, and other minor biases; \(iii\) Known defects —naive/cot/selfrefinestill carry residual Chinese placeholder fields in chapter slots under English loading mode, affecting 12 cells\.

#### The self\-preference bias introduced by the evaluator and generator sharing the same model is not mitigated in this study\.

All “winning” claims should therefore be treated as directional patterns rather than statistically significant conclusions; follow\-up experiments will introduce additional LLMs as evaluators to disentangle self\-preference bias\.

### 3\.2Research Questions

We organize research questions along two evidence chains — outline\-side primary evidence \(direct evaluation of outline outputs\) and writing\-side auxiliary evidence \(evaluation of writing outputs expanded from outlines\)\. Four research questions are posed observational; causal attribution is left to future ablation work\.

- •RQ1: Across the 7 frameworks, does a stable ranking emerge on outline\-side metrics? Do structured\-outline frameworks systematically outperform non\-outline baselines in faithfulness and consistency?
- •RQ2: Does the outline\-side ranking transfer to writing\-side metrics? Does SuperWriter remain first in the length\-constrained single\-chapter mode?
- •RQ3: On the same cells, what is the Spearman correlation between outline\-side and writing\-side rankings? Does it support the claim of outline–writing decoupling?
- •RQ4: Across 7 metric classes and the LLM\-as\-judge primary metrics, do cross\-dimensional variance and mean co\-vary? Is the co\-variation direction related to whether the writing strategy embeds explicit length control?

All four questions are descriptive and observational, and can be answered directly by experiments\. Answering causal\-level questions requires component\-level ablation experiments, which we leave for future work\. Sample\-size limitations prevent us from directly addressing cross\-language and cross\-genre robustness; a complete “framework×\\timeslanguage×\\timesgenre” cross\-tabulation is outside the scope of this study\.

### 3\.3Datasets and Frameworks

#### Datasets

are managed uniformly within the experimental scope, covering Chinese and English novels and other genres\. Each sample corresponds to an\(x,L\)\(x,L\)pair, wherexxis treated as existing text rather than a user instruction\. Its presentation form is determined by the loading mode \(singletakes one chapter;wholebookconcatenates the entire book\)\.LLis explicitly declared by each framework’s writing strategy via thetarget\_charsfield in the prompt and participates in computing the length\-compliance score\.

Table 1:Subset composition and sample counts\.This dataset satisfies three requirements: \(i\) cross\-genre — xianxia / xuanhuan / urban / fantasy web novels, English classics and contemporary novels, and academic / business other genres; \(ii\) cross\-language — roughly balanced between Chinese and English; \(iii\) cross\-length — each novel has 10 chapters, single chapters range from hundreds to thousands of characters, and concatenated whole books can exceed10,00010\{,\}000characters\. Of the 300 total samples, writing\-side evaluation is compute\-constrained and takes only a subset; LLM\-as\-judge direct outline\-side evaluation takes approximately 89 samples\. Both evaluations cover the four cells of Chinese vs\. English×\\timesnovels vs\. other genres\. Writing\-side samples are a subset of outline\-track samples, preserving dual\-track alignment\.

#### Three loading modes\.

The experiment exposes three loading semantics through a unified interface:singleflattens the data into one sample per chapter;multichapkeeps each book’s 10\-chapter list;wholebookconcatenates them into a single long string\. The original frameworks assume the input is a user instruction, whereas our samples are existing text and lack explicit character / plot introductions — this gives rise to the three systematic side effects described in Section[3\.1](https://arxiv.org/html/2608.26177#S3.SS1)\. Detailed properties of the 6 writing samples are listed in Appendix[Appendix D Writing 6\-sample Properties](https://arxiv.org/html/2608.26177#Ax4), Table[13](https://arxiv.org/html/2608.26177#Ax4.T13)\.

#### Compared frameworks\.

We include 7 frameworks, divided into non\-outline baselines and outline\-driven frameworks based on whether an outline is used\. The skeletons of all frameworks are isomorphic to those in the original papers or official code\.

Thenon\-outline baselinesincludenaive\(a single LLM call that directly outputs long text\),cot\(chain\-of\-thought, injecting “let’s think step by step” before the single call\), andselfrefine\(self\-refinement, looping for 2 rounds under a “generate→\\rightarrowfeedback→\\rightarrowrevise” skeleton with no specified review dimension\)\.

Theoutline\-driven long\-form generation frameworksinclude:CogWriter\(two\-stage block\-level planning, draft\-plan→\\rightarrowrevision\-plan; the constraint set is not enabled on the execution path\);Dome\(Campbell’s five\-act theory generating a five\-act outline;single/multichaprun in summary mode,wholebookruns in the paper’s original legacy mode\);StoryWriter\(outline agent \+ planning agent, with an event\-graph structure; the reproduction implements only the outline and planning agents, with the writing agent handled at the boundary but not implemented\);SuperWriter\(6\-step initialization agent, generating natural\-language paragraphs with hard word\-count declarations; the reproduction reduces iteration rounds to 1 and the refinement phase is disabled by default\)\. The complete LLM call counts and outline\-form comparison are shown in Table[2](https://arxiv.org/html/2608.26177#S3.T2)\.

Table 2:Comparison of the comparative frameworks: number of LLM calls and outline format\.

### 3\.4Evaluation Protocol

Following the exploratory claims of Section[1](https://arxiv.org/html/2608.26177#S1), we divide the evaluation protocol into two tracks: the outline\-side directly evaluates each framework’s outline outputs; the writing\-side evaluates writing expanded from those outlines\. The two tracks correspond to the evidence partition for RQ1 and RQ2\.

#### Outline\-side direct evaluation\.

We propose an anchor\-based LLM\-as\-a\-judge protocol that pre\-positions three example outlines as semantic anchors before rating\. The anchor scale is integer11–1010; the source text serves as the 5\-point anchor:55means aligned with the source text,≥6\\geq 6indicates a high\-quality outline, and≤4\\leq 4indicates a degraded outline\. The anchor design serves two purposes: \(i\) it provides the LLM evaluator with a concrete calibration anchor, avoiding the mid\-range collapse that occurs without anchors; \(ii\) it makes the cases𝑅𝐸\>5\\mathit\{RE\}\>5and𝑅𝐸<5\\mathit\{RE\}<5naturally interpretable\. Under the extraction task, using the source text as the 5\-point anchor carries the semantics of “alignment with the source text’s plot intent,” rather than “alignment with the premise” — readers should note this semantic shift when comparing across papers\.

#### Drift dimension family\.

We formalize the “four elements of drift” into an evaluable dimension family:

- •A1\_coverage: The outline’s coverage of key plot points in the source text\. The idea behind this dimension comes from the HANNA evaluation adopted by the StoryWriter framework\.
- •A2\_faithfulness: The outline’s faithfulness to the source text’s plot intent — if the outline introduces characters / events / plots not present in the source text, the maximum score is capped at66\. This dimension continues the emphasis on plot consistency in works such as DOME and SuperWriter\.
- •A3\_consistency: Internal non\-contradiction in the outline, including character attributes, timeline, and causal\-chain consistency, aligned with the CH dimension in HANNA\.
- •A4\_relevance: Outline on\-topic — all nodes must be relevant to the source text’s theme; if\>30%\>30\\%of nodes deviate, the outline is judged≤4\\leq 4\. This dimension borrows from the query\-relevant scoring rubric adopted by WritingBench\.
- •D1\_cross\_chapter\_consistency/M1\_continuity\_with\_prior: Activated only in whole\-book / multi\-chapter mode, capturing non\-drift across chapters and reflecting the need for global consistency under long\-range dependencies\.
- •overall\_quality: An integrative judgment, with the heaviest weighting on C1 \+ A1, serving as a summary metric of the outline’s overall executability\.

#### Reliability estimation\.

We employ a5%5\\%human\-calibration sample \(4 samples\); the calibration metric is the Pearson correlation between judge scores and human scores; due to the small sample size, it serves only as a directional reference\. Future work will report Cohen’sκ\\kappaon≥30\\geq 30calibration samples\.

#### Guardrails\.

We set 6 guardrails, corresponding to standard strategies for mitigating LLM\-as\-judge bias: \(i\) vigilance test — first evaluating deliberately constructed low\-quality anchors to verify non\-leniency; \(ii\) reasoning\-field length cap at 60 characters, to avoid verbosity bias; \(iii\) named anchors \+ difference mode, to avoid anchor drift; \(iv\)meta\.label\_confidenceself\-reported confidence, to avoid self\-preference bias; \(v\) bidirectional prompts, randomizing the A/B framework order to mitigate position bias; \(vi\) multi\-judge voting, with majority voting added in the accompanying report to mitigate single\-judge systematic bias\.

#### Writing\-side evaluation\.

The protocol rates each framework\-mode cell’s writing output across 7 metric classes\.

Table 3:The seven evaluation metrics used in the writing\-side evaluation\.Both HANNA and WriteJudge use a00–1010scale\. HANNA’s RE\+CH dimensions constitute the two main bolded columns of the writing\-side main table in Section[4\.3](https://arxiv.org/html/2608.26177#S4.SS3); the remaining 10 dimensions are secondary, sorted independently to avoid cherry\-picking\. All LLM\-as\-judge calls uniformly usedeepseek\-v4\-flash— our evaluator and generator are the same model; self\-preference bias is not mitigated in this study, and all absolute values require calibration after cross\-model robustness checks\.

#### Relationship between primary and auxiliary evidence\.

The former answers “which framework’s outline is more correct,” and the latter answers “which framework expands into better writing\.” They share the source text but evaluate different objects — this is the key pivot of the RQ3 consistency analysis\. They use the same LLM evaluator to maintain framework consistency, but this also introduces the uncontrolled intermediate variable of the “expansion step”: expansion may introduce losses or gains in faithfulness, so the decoupling observed in the consistency analysis of Section[4\.4](https://arxiv.org/html/2608.26177#S4.SS4)may partly stem from differences in expansion fidelity\.

### 3\.5Reproducibility and Known Limitations

#### Reproducibility fidelity\.

The mechanism\-level skeletons of all 7 frameworks are isomorphic to those in the original papers or official code\. Each deviation is annotated by its belonging element in Appendix[Appendix A Framework Reproduction Deviation List \(Critical / Significant / Minor\)](https://arxiv.org/html/2608.26177#Ax1)\(prompt orchestration, intermediate representation, or writing strategy\)\. Section[4\.4](https://arxiv.org/html/2608.26177#S4.SS4), Table[4](https://arxiv.org/html/2608.26177#S3.T4)lists representative deviations with significant impact on experimental conclusions; the complete 40\+ deviations are documented in the accompanying technical report\[[1](https://arxiv.org/html/2608.26177#bib.bib27)\]\.

Table 4:Representative reproducibility divergences across frameworks\.We classify deviations into critical \(may cause core mechanism failure or ranking reversal\), significant \(may introduce systematic bias but does not affect relative rankings\), and minor \(affect only local details\) based on their impact on experimental conclusions\. The representative deviations listed above are all critical — this means that what we compare are implementations under the backward\-extraction setting, not the full pipelines reported in the original papers: the claim in Section[4\.3](https://arxiv.org/html/2608.26177#S4.SS3)that “SuperWriter ranks first on single\-chapter HANNA” needs to be re\-examined after the critical deviations are fixed\. Representative frameworks such as MoPS / Re3 / DOC / RecurrentGPT are not included in the explicit comparison; the external validity of our research questions has not been verified on those frameworks\.

### 3\.6Experimental Pipeline

The overall pipeline consists of five steps, which in sequence produce outlines, intermediate representations, writing outputs, and evaluation results\. Steps 1\.5 and 3 are two independent LLM\-as\-judge calls, corresponding to the outline\-side LLM\-as\-judge and writing\-side LLM evaluation, respectively\.

Algorithm 1Five\-step experimental pipeline\.0:Dataset

𝒟\\mathcal\{D\}, framework pool

\{ℱ1,…,ℱK\}\\\{\\mathcal\{F\}\_\{1\},\\dots,\\mathcal\{F\}\_\{K\}\\\}, mode set

ℳ=\{single,multichap,wholebook\}\\mathcal\{M\}=\\\{\\texttt\{single\},\\texttt\{multichap\},\\texttt\{wholebook\}\\\}, judge model

𝒥\\mathcal\{J\}
1:Step 1: Outline generation\.For each framework\-mode combination, invoke the LLM in the framework’s prompt orchestration order to produce the final outline\.

2:Step 1\.5: Outline\-side LLM\-as\-judge\.Score only the outline text \(not the expanded writing\) on 6–7 dimensions, covering A1–A4 \+ overall\_quality \+ D1/M1; primary\-evidence data is in the accompanying report\.

3:Step 2: Long\-form writing\.For each combination, feed the outline into the writing strategy to produce the corresponding long text\.

4:Step 3: Writing\-side LLM evaluation\.Use the LLM evaluator to rate each writing output on M5 \(HANNA RE\+CH main and EM/SU/CR/CX secondary, 10 dimensions in total\) and M6 \(WriteJudge 6 dimensions\)\.

5:Step 4: Automatic metrics and aggregation\.Compute M1 \(word count\), M2 \(

SlS\_\{l\}\), M3 \(Ent\-2\), Distinct\-2, and M7 \(CR\), and aggregate with the LLM ratings from Steps 1\.5 and 3 by framework and mode into a unified summary\.

#### Complexity\.

The pipeline has linear complexity in framework count×\\timesmode count×\\timessample count\. Each cell requires 7 LLM calls\.

## 4Experiments

We conduct experiments on a controlled multi\-framework comparison platform to characterize the behavior of long\-form generation frameworks across different task modes\. This chapter is organized “primary evidence first, auxiliary evidence later”: Section[4\.2](https://arxiv.org/html/2608.26177#S4.SS2)reports outline\-side core results; Section[4\.3](https://arxiv.org/html/2608.26177#S4.SS3)reports per\-mode auxiliary tables and writing\-side main tables; Section[4\.4](https://arxiv.org/html/2608.26177#S4.SS4)presents outline\-side / writing\-side consistency analysis centered on RQ3; Section[4\.5](https://arxiv.org/html/2608.26177#S4.SS5)summarizes directional heuristics and the decoupling principle\.

### 4\.1Experimental Setup

This chapter’s experiments uniformly adopt the datasets and compared frameworks described in Section[3\.3](https://arxiv.org/html/2608.26177#S3.SS3)\. The writing\-side evaluation selects 6 samples to cover the cross of two languages and two genres; all 7 frameworks×\\times3 granularities form 21 comparison cells\. The three generation modes \(single/multichap/wholebook\), together with the length, n\-gram diversity, and writing\-quality metrics under the evaluation protocol in Section[3\.4](https://arxiv.org/html/2608.26177#S3.SS4), constitute the specific execution setup\. The reported experimental results characterize each framework’s behavior under a specific outline\-extraction task only; they are not directly comparable to the numbers reported in the original papers for the “premise→\\rightarrowlong text” forward generation task\.

### 4\.2Outline\-side Main Results

Table[5](https://arxiv.org/html/2608.26177#S4.T5)reports the means of 7 frameworks on 8 core dimensions across the three loading modes — one framework per row, one dimension per column, with cell values being the arithmetic mean across the three modes; per\-mode data aggregated by “book” and standard deviations are in the accompanying report and Appendix[Appendix B Per\-mode Per\-dimension Mean Tables \(B\.1–B\.4\)](https://arxiv.org/html/2608.26177#Ax2)\.

Table 5:7 frameworks on outline\-side primary metrics \(three\-mode per\-cell mean,n=3n=3–66per mode\)\. 1–5 integer scale; the overall score is the arithmetic mean of 8 dimensions\. Bold indicates the best in the column\. Per\-mode data is in Appendix[Appendix B Per\-mode Per\-dimension Mean Tables \(B\.1–B\.4\)](https://arxiv.org/html/2608.26177#Ax2)\.#### Observations

\(based on the mean trends above; directional observations, no significance tests performed\):

#### Headline conclusion\.

Structure\-driven frameworks systematically outperform non\-outline baselines on the four faithfulness dimensions \(A2 / A3 / A4\), and SuperWriter leads on 6 of the 8 dimensions\.

#### First

, the overall outline\-side advantage of structure\-driven frameworks\. On the four faithfulness dimensions \(A2 / A3 / A4\), structure\-driven frameworks show a stable advantage over non\-outline baselines: A2 faithfulness rises from3\.423\.42\(naive\) to4\.434\.43\(SuperWriter\), an improvement of about1\.01\.0point; A3 consistency rises from3\.393\.39to4\.714\.71, an improvement of about1\.31\.3points\. This trend holds across the three per\-mode tables, indicating a stable cross\-mode advantage\.

#### Second

, SuperWriter’s cross\-dimensional lead\. Across the 8 dimensions, SuperWriter ranks first on 6 of them \(A1 / A2 / A3 / A4 / B1 / C1\); Dome leads only on B2 hierarchy \(4\.244\.24vs\. SuperWriter3\.463\.46\)\. On the overall column, SuperWriter \(4\.364\.36\) leads the second\-place framework \(CogWriter3\.653\.65\) by about0\.70\.7points\. This lead is consistent with its writing\-side HANNA overall lead,suggesting that outline\-side advantage may transfer to writing\-side quality— a hypothesis we test in Section[4\.4](https://arxiv.org/html/2608.26177#S4.SS4)\.

#### Third

, the overall disadvantage of direct\-generation baselines\.naiveranks last or second\-to\-last across all 8 dimensions;cotandselfrefineimprove slightly on A3 / A4, but their overall scores remain significantly lower than structure\-driven frameworks \(2\.952\.95–3\.033\.03vs\.3\.553\.55–4\.364\.36\)\.

#### Fourth

, local trade\-offs for StoryWriter and CogWriter\. Among structure\-driven frameworks, StoryWriter ranks second on C1 executability \(4\.104\.10\); CogWriter scores lower on B2 hierarchy \(2\.862\.86\), reflecting the weakness of its block\-level planning intermediate representation in cross\-layer planning — consistent with the A\-1 deviation in Section[3\.5](https://arxiv.org/html/2608.26177#S3.SS5)\(constraint set not enabled, writing strategy degrades to “narrative polish”\)\.

#### Auxiliary tables\.

The overall\-quality means and standard deviations aggregated by “independent sample” are in the accompanying report; the difference between per\-mode means and the three\-mode mean is<0\.3<0\.3points and does not change the ranking order — SuperWriter still ranks first,naivestill ranks last\.

### 4\.3Per\-mode Analysis and Writing\-side Profile

#### Output length \(auxiliary table\)\.

Direct\-generation frameworks \(naive/cot/selfrefine\) produce the shortest outputs in single mode, but expand44–6×6\\timeswhen switching to wholebook\. Structure\-driven frameworks \(StoryWriter, Dome\) expand even more under wholebook \(StoryWriter expands from4,1424\{,\}142characters to28,30728\{,\}307characters, a6\.8×6\.8\\timesexpansion\), indicating that multi\-stage planning itself does not restrict output length\. SuperWriter produces the longest and most variance\-laden output in single mode \(single±4,864\\pm 4\{,\}864/ multichap±4,275\\pm 4\{,\}275/ wholebook±16,516\\pm 16\{,\}516; all three standard deviations are the highest among the 7 frameworks\), reflecting the combined effect of explicit word\-count declarations and the “full expansion” mechanism\.

Table 6:Output length \(in characters\) of the 7 frameworks by mode\. Each cell reports mean±\\pmstandard deviation; arranged in ascending order by the single column\. Sorting of wholebook / multichap may differ from single\.
#### n\-gram diversity \(auxiliary\)\.

Ent\-2 generally rises with scale \(10\.76→13\.2310\.76\\rightarrow 13\.23\), while Distinct\-2 generally decreases \(0\.840→0\.5520\.840\\rightarrow 0\.552\)\. Yet length is not the sole determining factor — Dome and SuperWriter produce nearly identical lengths under wholebook \(19,89619\{,\}896vs\.19,83519\{,\}835\), yet their Ent\-2 differs by0\.450\.45\(12\.7512\.75vs\.12\.3012\.30\), indicating that framework mechanisms and chapter structure also significantly affect diversity metrics\. The Pearson correlation between\|y\|\|y\|and Ent\-2 \(21 cell\-level data points\) is approximately0\.680\.68\(moderate positive correlation, but with substantial unexplained variance\)\. This phenomenon aligns with the known dependence of 2\-gram statistics on vocabulary size, indicating that within this length range, diversity metrics cannot be interpreted independently of length\.

#### Writing\-side multi\-dimensional profile \(main table\)\.

Table[7](https://arxiv.org/html/2608.26177#S4.T7)reports the means and standard deviations of two LLM evaluators over 6 writing samples, combining the three modes single / multichap / wholebook\.

Table 7:Quality of the 7 frameworks on HANNA 6 dimensions and WriteJudge overall on the writing side\. Each cell reports the mean±\\pmstandard deviation across 3 modes×\\times3 samples \(n=9n=9\)\. Bold indicates the best in each column\. Per\-mode means are in Appendix[Appendix B Per\-mode Per\-dimension Mean Tables \(B\.1–B\.4\)](https://arxiv.org/html/2608.26177#Ax2), Table[10](https://arxiv.org/html/2608.26177#Ax2.T10)\.
#### Main\-table observations:

#### First

, SuperWriter’s dual\-list lead\. SuperWriter ranks in the top 1 or top 2 on both HANNA overall \(6\.506\.50\) and WriteJudge overall \(6\.616\.61\) — first on HANNA, second on WriteJudge \(only0\.090\.09behindnaive’s6\.706\.70, a directional pattern\) — and is the only framework in the top 2 on both lists\. This result is consistent with its outline\-side overall lead, supporting the directional observation that “outline\-side advantage transfers to writing\-side quality\.”

#### Second

, SuperWriter’s specific\-dimension advantages\. On HANNA, SuperWriter ranks first on empathy EM \(7\.37\.3\), creativity CR \(7\.37\.3\), and complexity CX \(6\.86\.8\); on WriteJudge, it ranks first on creativity \(7\.87\.8\), specificity \(8\.08\.0\), and tone \(5\.95\.9\)\. The explicit per\-paragraph hard word\-count declaration penetrates the writing strategy through the prompt, allowing the writing stage to allocate paragraphs according to the declared length and thereby reducing the incidence of the two typical problems: “short truncation” and “repetition\.”

#### Third

,naive’s lack of systematic difference on WriteJudge\.naiveranks first on WriteJudge overall \(6\.70±1\.376\.70\\pm 1\.37\) but second on HANNA overall \(6\.07±1\.596\.07\\pm 1\.59\), a dual\-list ranking difference of 1\.naiveranks first on WriteJudge clarity \(7\.47\.4\) and coherence \(6\.96\.9\), reflecting that the “free expansion” characteristic of the “no planning” baseline shows no systematic difference from structure\-driven frameworks under WriteJudge evaluation \(directional pattern\)\.

#### Fourth

, the uneven performance of structure\-driven frameworks\. StoryWriter \(HANNA5\.745\.74/ WJ6\.196\.19\), CogWriter \(HANNA5\.695\.69/ WJ6\.006\.00\), and Dome \(HANNA5\.575\.57/ WJ6\.086\.08\) cluster in the middle range, with no significant lead\. Their mode\-specific advantages \(StoryWriter’s event\-driven narrative in single / Dome’s Campbell five\-act structure in wholebook are useful\) are masked in the three\-mode aggregated mean\.

#### Per\-mode breakdown\.

Table[8](https://arxiv.org/html/2608.26177#S4.T8)reports the means and standard deviations of the two evaluators across the three modes\.

Table 8:Comparison of writing\-side quality across modes on HANNA and WriteJudge overall for the 7 frameworks\. Mean±\\pmstandard deviation \(n=3n=3per cell\)\. Arranged in descending order by the HANNA single column\. SuperWriter ranks first on HANNA single, and second on WJ single \(0\.230\.23behindnaive, a directional pattern\)\.As shown in Table[8](https://arxiv.org/html/2608.26177#S4.T8), explicit length constraint ranks first under the most length\-constrained setting\. Under single mode, SuperWriter simultaneously achieves the highest HANNA \(6\.836\.83\) and the second\-highest WriteJudge \(6\.946\.94\), leading with relatively low standard deviations \(0\.760\.76/0\.630\.63\)\. This advantage is most pronounced in the most length\-constrained single mode, but degrades under wholebook \(5\.895\.89\), at which point explicit paragraph budgets give way to the long\-range coherence challenges common to all frameworks\.

The uneven performance of structure\-driven frameworks\. StoryWriter’s multi\-event design is best suited for event\-driven narrative under single \(HANNA6\.226\.22, WriteJudge6\.616\.61\), but degrades under multichap \(WriteJudge5\.445\.44\), indicating that its per\-chapter event allocation steps achieve internal optimum at the cost of cross\-chapter coherence\. Dome’s Campbell five\-act structure is most useful for whole\-book macro\-structure, but contributes limited perceived quality under single \(HANNA5\.945\.94, WriteJudge6\.586\.58, middle range\)\.

### 4\.4Outline–writing Consistency Analysis

#### Outline–writing consistency\.

This section answers RQ3 — examining the coupling and decoupling between outline\-side LLM\-judge direct\-evaluation rankings and writing\-side HANNA / WriteJudge rankings\. Specific Spearman correlation values are detailed in Appendix[Appendix B Per\-mode Per\-dimension Mean Tables \(B\.1–B\.4\)](https://arxiv.org/html/2608.26177#Ax2), Table[12](https://arxiv.org/html/2608.26177#Ax2.T12)\.

#### Overall consistency\.

The outline faithfulness ranking shows a moderate positive correlation with the writing\-side HANNA RE\+CH ranking \(Spearmanρ≈0\.45\\rho\\approx 0\.45–0\.580\.58\): frameworks with higher outline faithfulness also have higher writing RE\+CH, but the writing\-side EM/SU/CX advantages can be independently achieved by SuperWriter through explicit per\-paragraph word\-count declarations\.

#### Three deviation patterns\.

We observe three typical deviation patterns: \(1\) “dual\-high” pattern — high outline A2\_faithfulness and high writing HANNA RE, occurring in frameworks with structured outlines \+ multi\-chapter fine\-grained planning; \(2\) “trading length for polish” pattern — large writing\-length variance and high WriteJudge / low HANNA RE, with SuperWriter as the typical case, showing that explicit per\-paragraph hard word\-count declarations enable “full expansion” on the writing side to simultaneously boost multiple dimensions \(including EM/SU/CX\), but do not necessarily boost outline\-side faithfulness in sync; \(3\) “reverse” pattern — low outline A2\_faithfulness but high writing WriteJudge, typically manifested in the behavior of multi\-stage frameworks under wholebook\.

#### Intermediate variable and observation boundary\.

We do not control expansion fidelity as an independent dimension; the expansion step may introduce losses or gains in faithfulness, so the decoupling observed in the consistency analysis may partly stem from losses / gains in expansion fidelity rather than purely from “outline–writing ranking decoupling” — this uncontrolled intermediate variable is an open limitation of the paper\. Based on the two factors above \(self\-preference bias and uncontrolled expansion fidelity\), the directional nature of the “trade\-off” mechanism in this section should be treated as an exploratory claim and subjected to rigorous validation after larger samples and cross\-model robustness checks\.

#### Sensitivity analysis\.

In view of the known critical deviations described in Section[3\.5](https://arxiv.org/html/2608.26177#S3.SS5)\(SuperWriter’s refinement phase disabled by default, CogWriter’s constraint set not enabled on the execution path\), we exclude these two frameworks from the per\-mode samples and recompute the rankings: before exclusion, the single\-column ranking is SuperWriter\>\>StoryWriter\>\>CogWriter\>\>Dome\>\>selfrefine\>\>cot\>\>naive; after exclusion, the ranking is StoryWriter \> Dome \> selfrefine \> cot \> naive — both SuperWriter and CogWriter are excluded, and the top rank is taken over by StoryWriter\. This sensitivity analysis shows that the claim “SuperWriter ranks first on single\-chapter HANNA” depends strongly on these critical\-deviation frameworks; after exclusion, the claim no longer holds\.

### 4\.5Heuristic Observations and Decoupling Principle

#### Three observations emerge\.

#### First

, framework performance depends primarily on the match between its intrinsic output form and the target granularity — SuperWriter, with its hard length constraint, performs best in single mode where length control is the tightest constraint \(HANNA6\.836\.83\); its paragraph\-list per\-paragraph hard word\-count declaration provides a constraint interface for the writing stage;naiveandcot, lacking such constraints, lose the least when the task expands to wholebook\.

#### Second

, explicit constraints together with the “full expansion” mechanism jointly cause variance in writing length — SuperWriter has the highest length variance among the 7 frameworks, indicating that this constraint should be understood as a composition of two opposing sub\-propositions \(quality variance and length variance\), avoiding one\-directional rhetoric of “explicit is better than implicit\.”

#### Third

, writing\-quality scores that are consistent across dimensions tend to be higher — SuperWriter has the smallest cross\-dimensional standard deviation,selfrefinethe largest; this is an arithmetic consequence of Jensen’s inequality \(given a fixed mean ofnndimensions, the smaller the cross\-dimensional variance, the larger the gap between the mean and the minimum\), and is independent of experimental verification\.

#### Ranking stability\.

Cross\-framework relative rankings are unstable across generation granularities: StoryWriter’s WriteJudge overall score ranges from5\.445\.44\(multichap\) to6\.616\.61\(single\); SuperWriter drops from6\.946\.94\(single\) to6\.336\.33\(multichap\);naiverises from6\.286\.28\(multichap\) to6\.676\.67\(wholebook\)\. No framework can dominate all three granularities; long\-form evaluation should report at least single and wholebook granularities together, with multichap as an intermediate mode\.

#### Decoupling principle\.

Based on the observations and consistency analysis above, we propose the following decoupling principle: outline planning and outline expansion in long\-form generation should be treated as two separable engineering stages, each with its own metric system and evaluation\. The causal mechanism of this principle awaits future verification — attempting to have the outline stage simultaneously carry emotional injection \(EM\) and creative tension \(SU/CR\) may trade off against faithfulness goals; within the scope of our observations, leaving the latter to the expansion or post\-processing stage is a more reasonable design choice\.

## 5Discussion

This chapter situates the experimental results of this study against the original work of existing long\-form generation frameworks\. Our controlled experiments show that structure\-driven frameworks systematically outperform non\-outline baselines on the faithfulness dimensions \(A2 / A3 / A4\): A2 faithfulness rises fromnaive’s3\.423\.42toSuperWriter’s4\.434\.43, an improvement of about1\.01\.0points; A3 consistency rises from3\.393\.39to4\.714\.71, an improvement of about1\.31\.3points\. This trend aligns with the quality improvement brought by structured planning reported in the original papers — CogWriter achieves instruction\-completion accuracy superior to GPT\-4o on LongGenBench with Qwen\-2\.5\-14B as the backend through its planning agent’s two\-stage block\-level decomposition; DOME simultaneously improves Ent\-2 \(6\.3%6\.3\\%–35\.7%35\.7\\%gain\) and reduces the conflict rate \(15\.2%15\.2\\%–27\.3%27\.3\\%reduction\) over DOC and Re3baselines through its dynamic hierarchical outline mechanism; StoryWriter achieves the best mean of4\.24\.2from both human and automatic ratings on MoPS through its event graph and consistency\-check modules\. However, the magnitude of the relative advantage we observe \(about11point\) is significantly smaller than the improvements reported in the original papers, possibly for these reasons: the original papers mostly evaluate the forward “premise→\\rightarrowlong text” task, while we adopt the backward\-extraction task, where the source text is already a high\-quality finished product, imposing a ceiling effect on outline faithfulness; the original papers’ comparison baselines are mostly single\-prompt or CoT, whereas we additionally includeselfrefine; the self\-preference bias of evaluator and generator sharing the same model may affect structure\-driven frameworks asymmetrically\.

Different frameworks have advantages that vary with task granularity\. SuperWriter ranks first in the length\-constrained single\-chapter mode \(HANNA6\.836\.83\), but this advantage degrades in whole\-book mode \(HANNA5\.895\.89\); this observation contrasts with SuperWriter’s original paper’s reported strong results of an overall WritingBench score of8\.518\.51and a real\-query win rate exceeding98%98\\%— the original paper focuses on the short\-prompt scenario, while this study’s whole\-book mode requires generating continuous text exceeding10,00010\{,\}000characters, with explicit paragraph budgets giving way to the long\-range coherence challenges common to all frameworks\. Dome’s Campbell five\-act structure is most useful for whole\-book macro\-structure, but contributes limited perceived quality under single\-chapter mode \(HANNA5\.945\.94\); StoryWriter’s multi\-event design is best suited for event\-driven narrative under single \(HANNA6\.226\.22\), but degrades under multichap \(WriteJudge5\.445\.44\) — the dynamic outline mechanism is effective at the macro\-structure level, while multi\-event planning is effective at the local narrative level, each with shortcomings at the cross\-chapter granularity\. Notably,naive’s dual\-list discrepancy of ranking first on WriteJudge overall \(6\.706\.70\) but second on HANNA overall \(6\.076\.07\), as well as the moderate ranking correlation of HANNA and WriteJudge across the 7 frameworks, both suggest that cross\-metric\-system compatibility itself is a research object of evaluation methodology; a single metric system cannot fully characterize framework capabilities\.

The relative position of thenaivebaseline depends heavily on task setting and metric choice: CogWriter’s original paper shows thatnaive\-class single\-generation models have an extremely low completion rate \(0\.460\.46\) on LongGenBench, while StoryWriter’s original paper shows that stories directly generated by GPT\-4o\-mini average only1,0781\{,\}078characters; yet in this studynaiveanomalously leads on WriteJudge — the difference mainly stems from the evaluation metric system \(WriteJudge is more lenient toward the “free expansion” characteristic of “no\-planning” baselines\), the task\-inversion setting \(naivedirectly facing source\-text extractive generation has higher information density\), and the samples being primarily mid\- and short\-chapter without sufficiently covering ultra\-long tasks\. Readers should strictly align tasks and metrics when comparing across papers\.

Finally, this study adopts the backward “long text→\\rightarrowoutline” extraction task, which differs systematically from the forward “premise→\\rightarrowlong text” generation task commonly adopted by the original papers: there is a ceiling effect on outline faithfulness; some framework\-specific mechanisms \(SuperWriter refinement, CogWriter constraint set\) are difficult to trigger under the backward task; the unmitigated evaluator–generator self\-preference bias makes all “winning” claims directional patterns\. This caveat does not deny the methodological significance of the phenomenon that outline\-side and writing\-side rankings correlate only moderately across task settings\. Based on the above comparative analysis, we offer three implications for framework design and benchmark construction: structure\-driven frameworks should focus design on simultaneously improving outline faithfulness and consistency, while emotional injection and creative tension are better suited to the expansion or post\-processing stage; explicit length constraints should be viewed as a “trade\-off between quality variance and length variance” rather than one\-directional rhetoric of “explicit is better than implicit”; subsequent benchmarks should include dual\-track evaluation on both the outline side and the writing side, and report cross\-metric\-system compatibility\.

## 6Conclusion

We conducted a systematic head\-to\-head comparison of 7 long\-form generation frameworks across 3 generation granularities \(single\-chapter / multi\-chapter / whole\-book\) and 7 metric classes on a unified controlled platform\.Two key observations emerge\.First, in the most length\-constrained single\-chapter mode, SuperWriter ranks first on the HANNA overall score \(6\.836\.83\), but this advantage is jointly affected by SuperWriter’s refinement phase being disabled by default and self\-preference bias — after critical\-deviation frameworks are excluded, the claim no longer holds; no stable global framework ranking across task modes exists \(StoryWriter’s event\-driven narrative is effective under single but degrades under multichap; Dome’s Campbell five\-act structure is most useful for whole\-book macro\-structure but contributes limited under single\)\. Second, the outline\-side and writing\-side rankings correlate moderately positively \(Spearmanρ≈0\.45\\rho\\approx 0\.45–0\.580\.58\), supporting the outline–writing decoupling principle — writing\-side metrics should not be used to backward\-infer outline\-side performance\. Compute constraints limit the writing\-side sample size; follow\-up experiments will expand the sample size and add additional models as evaluators for controlled comparisons\. Moreover, this study adopts the backward\-extraction task rather than the forward\-generation task reported in the original papers, and the task\-setting difference should be noted when comparing across papers\.

## 7Ethical Statement

This research does not involve human subjects, personally identifiable information, or sensitive content\. All datasets used \(WritingPrompts subset, Chinese novels, other genres\) are publicly available\. This study uses the large language model \(deepseek\-v4\-flash\) solely as an evaluation model and generation backend, not as a content\-creation agent\. To the authors’ knowledge, no ethical concerns arise in this paper\.

## 8Code and Data Availability

This research has currently obtained stage\-by\-stage experimental results\. After the follow\-up experiments are completed and the code, configuration files, and per\-cell evaluation results are organized, the code, data, and complete test records will be released under the Apache 2\.0 license\. Pre\-trained model checkpoints will not be redistributed; all experiments use publicly available models \(deepseek\-v4\-flash\)\. The accompanying technical report\[[1](https://arxiv.org/html/2608.26177#bib.bib27)\]preserves the complete 12\-dimension per\-dimension means and cell\-level raw outputs\.

## References

- Authors \(2026\)A\. AuthorsFramework pipeline and reproducibility diff summary \(internal audit document\)\.Note:Project internal documentMaintained 2026\-08\-11Cited by:[Appendix A Framework Reproduction Deviation List \(Critical / Significant / Minor\)](https://arxiv.org/html/2608.26177#Ax1.p1.1),[Appendix C Cell\-level Per\-cell Data](https://arxiv.org/html/2608.26177#Ax3.p1.1),[§3\.5](https://arxiv.org/html/2608.26177#S3.SS5.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.26177#S8.p1.1)\.
- Baiet al\.\(2024\)Y\. Bai, J\. Zhang, X\. Lv, L\. Zheng, S\. Zhu, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongWriter: unleashing 10,000\+ word generation from long context llms\.Note:arXiv preprint arXiv:2408\.07055Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Chhunet al\.\(2022\)C\. Chhun, P\. Colombo, F\. M\. Suchanek, and C\. ClavelOf human criteria and automatic metrics: a benchmark of the evaluation of story generation\.Proceedings of the 29th International Conference on Computational Linguistics \(COLING 2022\), pp\. 5794\-5836\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Fanet al\.\(2018\)A\. Fan, M\. Lewis, and Y\. N\. DauphinHierarchical neural story generation\.ACL 2018, Melbourne, Australia, Volume 1: Long Papers, pages 889\-898\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.
- Goldfarb\-Tarrantet al\.\(2019\)S\. Goldfarb\-Tarrant, H\. Feng, and N\. PengPlan, write, and revise: an interactive system for open\-domain story generation\.Note:arXivCited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Jiang, M\. Wang, and X\. ZhangA survey on llm\-as\-a\-judge\.Note:arXiv preprint arXiv:2411\.15594Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1)\.
- Honget al\.\(2023\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. CaoMetaGPT: meta programming for a multi\-agent collaborative framework\.Note:arXiv preprint arXiv:2308\.00352Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Huotet al\.\(2024\)F\. Huot, R\. K\. Amplayo, J\. Palomaki, A\. S\. Jakobovits, E\. Clark, and M\. LapataAgents’ room: narrative generation through multi\-step collaboration\.Note:arXiv preprint arXiv:2410\.02603Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Linet al\.\(2025\)L\. Lin, H\. Wang, Z\. Li, and X\. WeiWebNovelBench: placing llm novelists on the web novel distribution\.Note:arXiv preprint arXiv:2505\.14818Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Note:arXiv preprintCited by:[§1](https://arxiv.org/html/2608.26177#S1.p1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Clark, and Y\. YangSelf\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1)\.
- Pinget al\.\(2025\)B\. Ping, J\. Zeng, F\. Meng, S\. Wang, J\. Zhou, and S\. ZhangLongDPO: unlock better long\-form generation abilities for llms via critiqued\-aware stepwise information\.Note:arXiv preprint arXiv:2502\.02095Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Queet al\.\(2024\)H\. Que, F\. Duan, L\. He, Y\. Mou, W\. Zhou, J\. Liu, W\. Rong, Z\. M\. Wang, J\. Yang, G\. Zhang, J\. Peng, Z\. Zhang, S\. Zhang, and K\. ChenHelloBench: evaluating long text generation capabilities of large language models\.Note:arXiv preprint arXiv:2409\.16191Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Note:arXiv preprint arXiv:2305\.18290Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Salemiet al\.\(2025\)A\. Salemi, J\. Killingback, and H\. ZamaniExPerT: effective and explainable evaluation of personalized long\-form text generation\.Note:arXiv preprint arXiv:2501\.14956Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Shaoet al\.\(2024\)Y\. Shao, Y\. Jiang, T\. A\. Kanell, P\. Xu, O\. Khattab, and M\. S\. LamAssisting in writing wikipedia\-like articles from scratch with large language models\.Proceedings of NAACL 2024, pages 6252\-6278\.Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InNeurIPS 2023,Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Tanet al\.\(2024\)H\. Tan, Z\. Guo, Z\. Shi, L\. Xu, Z\. Liu, Y\. Feng, X\. Li, Y\. Wang, L\. Shang, Q\. Liu, and L\. SongPROXYQA: an alternative framework for evaluating long\-form text generation with large language models\.Proceedings of ACL 2024, pages 6806\-6827\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Liu, J\. Zhu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. WangAutoGen: enabling next\-gen llm applications via multi\-agent conversation\.Note:arXiv preprint arXiv:2308\.08155Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024\)Y\. Wu, M\. S\. Hee, Z\. Hu, and R\. K\.\-W\. LeeLongGenBench: long\-context generation benchmark\.Proceedings of ICLR 2024\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p1.1)\.
- Wuet al\.\(2025a\)Y\. Wu, J\. Mei, M\. Yan, C\. Li, S\. Liang, Y\. Wang, Z\. Wang, J\. Zhang, M\. Wu, Q\. Jin, and F\. HuangWriteJudge: a multi\-aspect evaluator for generated text via calibrated language model with external knowledge\.arXiv preprint\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2025b\)Y\. Wu, J\. Mei, M\. Yan, C\. Li, S\. Liang, Y\. Wang, Z\. Wang, J\. Zhang, M\. Wu, Q\. Jin, and F\. HuangWritingBench: a comprehensive benchmark for generative writing\.Note:arXivCited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2023\)K\. Yang, D\. Klein, N\. Peng, and Y\. TianDOC: improving long story coherence with detailed outline control\.Proceedings of ACL 2023, pages 3378\-3465\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2022\)K\. Yang, Y\. Tian, N\. Peng, and D\. KleinRe3: generating longer stories with recursive reprompting and revision\.Proceedings of EMNLP 2022, pages 4393\-4479\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2019\)L\. Yao, N\. Peng, R\. Weischedel, K\. Knight, D\. Zhao, and R\. YanPlan\-and\-write: towards better automatic storytelling\.InProceedings of the 33rd AAAI Conference on Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\.\-L\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.NeurIPS 2023, New Orleans, LA, USA\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p3.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2023\)W\. Zhou, Y\. E\. Jiang, P\. Cui, T\. Wang, Z\. Xiao, Y\. Hou, R\. Cotterell, and M\. SachanRecurrentGPT: interactive generation of \(arbitrarily\) long text\.CoRR, abs/2305\.13304\.Cited by:[§1](https://arxiv.org/html/2608.26177#S1.p2.1),[§2](https://arxiv.org/html/2608.26177#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix A Framework Reproduction Deviation List \(Critical / Significant / Minor\)

This appendix reports all known deviations of the 7 compared frameworks from the original papers or official code during reproduction, classified by their impact on experimental conclusions into critical, significant, and minor\. The main text \(Section[3\.5](https://arxiv.org/html/2608.26177#S3.SS5)\) has referenced 3 representative critical deviations; the complete 40\+ deviations are in the accompanying technical report\[[1](https://arxiv.org/html/2608.26177#bib.bib27)\]\. This appendix provides placeholder descriptions only; the complete table is in the English\-version Appendix A\.

## Appendix B Per\-mode Per\-dimension Mean Tables \(B\.1–B\.4\)

This appendix reports the per\-dimension means and standard deviations of the outline\-side LLM\-judge direct evaluation across the three modes single / multichap / wholebook \(B\.1 / B\.2 / B\.3\), as well as the overall\-quality means and standard deviations aggregated by “independent sample” \(B\.4\)\. The main text \(Section[4\.2](https://arxiv.org/html/2608.26177#S4.SS2)\) has referenced the per\-mode per\-dimension raw data in B\.1–B\.4; this appendix provides placeholder descriptions only; the complete tables are in the English\-version Appendix B\.

Table 9:Outline\-side primary\-metric means \(n≈89n\\approx 89; A1–A4 \+ overall\_quality \+ D1/M1; 1–10 scale\)\. Cells marked with∗are affected by known defects in Section[3\.1](https://arxiv.org/html/2608.26177#S3.SS1)— the English\-side data of these cells may be degraded\.Table 10:Writing\-side track means \(samen=6n=6subset; HANNA 6 dimensions \+ WriteJudge overall; 1–10 scale\)\.Table 11:Dual\-list ranking difference \(HANNA overall vs\. WriteJudge overall;n≈89n\\approx 89\)\. “Dual\-list agreement” is operationally defined as ranking difference≤1\\leq 1; of the 21 cells, 12 cells show dual\-list agreement, 9 cells show disagreement\.Table 12:Outline vs\. Writing Spearmanρ\\rho\(n≈89n\\approx 89; primary\-evidence overall\)\. Allρ\\rhointerpretations should be treated with caution — self\-preference bias is not mitigated; expansion fidelity is uncontrolled as an intermediate variable\.Dimension Pairρ\\rhoInterpretationA2\_faithfulness \(outline\) vs\. HANNA RE \(writing\)0\.450\.45Weak\-moderate positive correlationA3\_consistency \(outline\) vs\. HANNA CH \(writing\)0\.520\.52Weak\-moderate positive correlationA4\_relevance \(outline\) vs\. WJ relevance \(writing\)0\.580\.58Weak\-moderate positive correlationoverall\_quality \(outline\) vs\. WJ overall \(writing\)0\.430\.43Weak\-moderate positive correlation\(EM, A2\_faithfulness\)−0\.32\-0\.32Weak negative correlation \(trade\-off\)\(SU, A2\_faithfulness\)−0\.28\-0\.28Weak negative correlation \(trade\-off\)\(CR, A2\_faithfulness\)−0\.21\-0\.21Weak negative correlation \(trade\-off\)overall\_quality vs\. A1\_coverage0\.780\.78Strong positive correlationoverall\_quality vs\. A2\_faithfulness0\.650\.65Moderate positive correlationoverall\_quality vs\. A3\_consistency0\.620\.62Moderate positive correlationoverall\_quality vs\. A4\_relevance0\.710\.71Moderate\-strong positive correlation
## Appendix C Cell\-level Per\-cell Data

This appendix provides placeholder descriptions only; the complete cell\-level data for 63 rows \(= 9 cells×\\times7 frameworks\) is preserved in the accompanying technical report\[[1](https://arxiv.org/html/2608.26177#bib.bib27)\]and not duplicated here\. The cell\-level data covers single mode \(samplesnovel\_ch\_idx15,novel\_en\_idx4,other\_idx32\), multichap mode \(samplesnovel\_ch\_idx15,novel\_ch\_idx41,novel\_en\_idx55\), and wholebook mode \(samplesnovel\_ch\_idx15,novel\_ch\_idx41,novel\_en\_idx4\)\. The remaining 9 cells were not run due to compute constraints;other\_idx76was excluded due to lack of wholebook data\. For each cell, the data point includes word count, Ent\-2, Distinct\-2, length\-compliance scoreSqS\_\{q\}, and WriteJudge overall\.

## Appendix D Writing 6\-sample Properties

This appendix lists the detailed properties of the 6 samples used in the writing\-side evaluation in Section[4\.1](https://arxiv.org/html/2608.26177#S4.SS1), for reviewer verification\.

Table 13:Writing 6\-sample properties\.#### 6\-sample selection criteria

: \(i\) Each of the four cells \(Chinese, English\)×\\times\(novel, other\) has≥1\\geq 1case — among the 100 other\-genre samples, 50 Chinese \+ 50 English, with the 6 samples each taking 1 \(idx32/idx76\); \(ii\) The 2novel\_chcases include 1 long\-context case \(idx41\) to examine the long\-context scenario; \(iii\) Sampling is convenience sampling rather than random sampling\.