Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

# A Reader-Centered Axis for Summarization Evaluation
Source: [https://arxiv.org/html/2608.14457](https://arxiv.org/html/2608.14457)
## Information Satisfaction: A Reader\-Centered Axis for Summarization EvaluationConference:2026 ACM Conference on Human\-AI Complementarity and Alignment; September 27–30, 2026; Alexandria, VA, USA2026 ACM Conference on Human\-AI Complementarity and Alignment \(HCOMP 2026\), September 27–30, 2026, Alexandria, VA, USADOI:[10\.1145/3834580\.3838749](https://doi.org/10.1145/3834580.3838749)ISBN:979\-8\-4007\-2894\-5/2026/09CCS:Information systems SummarizationCCS:Human\-centered computing User studiesCCS:Computing methodologies Natural language processingCCS:Computing methodologies Machine learning

Isabel CacholaAffiliation:St\. Edward’s University,Austin,TX,USANote:Work conducted at Johns Hopkins University\. Email: icachola@stedwards\.eduWilliam WaldenAffiliation:Johns Hopkins University,Human Language Technology Center of Excellence,Baltimore,MD,USA,Reno KrizAffiliation:Johns Hopkins University,Human Language Technology Center of Excellence,Baltimore,MD,USAandMark DredzeAffiliation:Johns Hopkins University,Baltimore,MD,USA

2026; © cc

###### Abstract\.

The majority of work on summarization evaluation focuses on general summary quality \(e\.g\., ROUGE, BERTScore\) or specific desired properties \(e\.g\., readability, factuality\)\. However, these metrics fail to measure the utility of a summary to an individual user\. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor\. Query\-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician\. By contrast, a reader’s background or*persona*—their role and expertise—is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader’s needs\. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM\-as\-judge metrics, fail basic perturbation tests of informational content\. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person’s background and use case\. We find that both traditional and LLM\-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment\.

###### Keywords:

summarization evaluation, query\-focused summarization, information satisfaction, reader personas, evaluation metrics, LLM\-as\-judge, human evaluation, metric correlation, user\-centered NLP

††cc\-license:by## 1\.Introduction

The value of a summary is rarely intrinsic; rather, it is realized only when the summary helps a particular reader accomplish a particular goal\. Yet automatic summarization has long been evaluated through metrics that aim to capture a general notion of quality as a property of the summary itself\. Lexical overlap measures like ROUGE compare candidate summaries against reference texts\([36](https://arxiv.org/html/2608.14457#bib.bib1)\); embedding\-based approaches like BERTScore assess semantic similarity\([63](https://arxiv.org/html/2608.14457#bib.bib4)\); and more targeted metrics evaluate specific dimensions such as factual consistency, coherence, or readability\([31](https://arxiv.org/html/2608.14457#bib.bib44);[59](https://arxiv.org/html/2608.14457#bib.bib8);[13](https://arxiv.org/html/2608.14457#bib.bib32);[41](https://arxiv.org/html/2608.14457#bib.bib29)\)\. More recently, Large Language Models \(LLMs\) have been deployed as judges, promising more nuanced assessments that approach human\-level evaluation\([65](https://arxiv.org/html/2608.14457#bib.bib14);[38](https://arxiv.org/html/2608.14457#bib.bib11)\)\. Across this proliferation of metrics, however, quality is treated as a fixed attribute that can be measured against a gold standard or set of reference criteria, independent of who is reading\.

This framing obscures a fundamental truth about summarization: a summary is only useful insofar as it serves the informational needs of the person reading it\. Consider a recent paper on mRNA vaccine development\. A biomedical researcher may need a summary that preserves methodological detail, statistical results, and connections to prior literature, while a family physician reading the same paper may be best served by a summary emphasizing clinical implications, patient guidance, and contraindications\. The source document is the same in each case, and a reference summary written for one audience may score highly under standard metrics while wholly failing to satisfy another reader’s needs\. Most existing evaluation paradigms have little to say about this mismatch because they either do not model the reader at all\([36](https://arxiv.org/html/2608.14457#bib.bib1);[63](https://arxiv.org/html/2608.14457#bib.bib4)\)or focus on stylistic preferences of a reader, rather than informational\([38](https://arxiv.org/html/2608.14457#bib.bib11);[61](https://arxiv.org/html/2608.14457#bib.bib66)\)\.

To capture this reader\-centered notion of utility, we introduceinformation satisfactionas an axis of summarization evaluation\. Information satisfaction measures the extent to which a summary resolves the specific informational need that motivated the reader to consult the source in the first place—operationalized in this work as a query paired with a*persona*describing the reader’s role and expertise\. In theory, a sufficiently detailed query could encode everything about the reader, negating the need for the persona\. In practice, however, readers rarely provide such information in their requests, leaving the reader’s background, expertise, and purpose implicit\. A persona recovers exactly this missing context, and because a reader’s role and expertise are comparatively stable across the many queries they issue, it can be specified once and reused rather than re\-elicited for every request\. We therefore treat the query and the persona as complementary, with the query expressing the immediate question and the persona supplying the relatively static reader context that the query leaves unsaid\. A summary can be fluent, factually accurate, and a faithful condensation of its source while still leaving the reader’s needs unaddressed\. Conversely, a stylistically rough summary may fully satisfy the query at hand for a reader with the right background\. By foregrounding both the query and the reader behind it, information satisfaction reframes evaluation around the reader’s purpose rather than the summary’s surface properties\.

Given this reframing, we ask whether existing summarization metrics are equipped to measure information satisfaction\. We organize our investigation around two research questions:

#### RQ1Are evaluation metrics sensitive to variation in informational content and reader persona?

A metric that meaningfully tracks information satisfaction should respond when the information in a summary shifts relative to what the reader needs—either because the summary’s content changes or because the persona consuming it changes\. We probe this property by systematically varying both axes and measuring how popular automatic metrics, including traditional reference\-based measures and LLM\-as\-judge approaches, respond to the perturbations\.

#### RQ2Do evaluation metrics agree with human judgments of information satisfaction?

Even a metric that is sensitive to content and persona changes is only useful if its judgments align with those of real readers\. We conduct an expert human evaluation in which annotators rate summaries with respect to a specified query and persona, and we measure the correlation between these human judgments and the scores produced by automatic metrics\.

Here, we show that many popular metrics, including strong LLM\-as\-judge metrics, fail simple perturbation tests to the informational content of a summary\. It is thus unsurprising that we further observe that the same metrics show poor agreement with human judges of information satisfaction\. Taken together, our results suggest that neither traditional nor LLM\-based metrics adequately capture information satisfaction, and that progress on user\-centered summarization requires evaluation frameworks that explicitly model who the summary is for and what they seek to learn\.

## 2\.Related Works

Automatic summarization evaluationhas historically centered onnn\-gram overlap with reference summaries, most notably ROUGE\([36](https://arxiv.org/html/2608.14457#bib.bib1)\)\. While simple and reproducible, ROUGE rewards surface overlap rather than semantic equivalence and correlates only weakly with human judgments of quality\([37](https://arxiv.org/html/2608.14457#bib.bib2);[4](https://arxiv.org/html/2608.14457#bib.bib3)\)\. Embedding\-based metrics such as BERTScore\([63](https://arxiv.org/html/2608.14457#bib.bib4)\)and MoverScore\([64](https://arxiv.org/html/2608.14457#bib.bib5)\)address this by comparing contextualized representations, and learned metrics like BLEURT\([50](https://arxiv.org/html/2608.14457#bib.bib6)\)train directly on human ratings\. A parallel line of work targets specific aspects of summary quality, particularly factual consistency, with metrics including FactCC\([32](https://arxiv.org/html/2608.14457#bib.bib7)\), QAGS\([59](https://arxiv.org/html/2608.14457#bib.bib8)\), QuestEval\([48](https://arxiv.org/html/2608.14457#bib.bib9)\), and SummaC\([34](https://arxiv.org/html/2608.14457#bib.bib10)\)\. More recently, large language models have been used directly as evaluators of summary quality and have been shown to outperform earlier automatic metrics on several aspects of summary evaluation\([38](https://arxiv.org/html/2608.14457#bib.bib11);[15](https://arxiv.org/html/2608.14457#bib.bib12);[7](https://arxiv.org/html/2608.14457#bib.bib13)\)\. These metrics share a common assumption that summary quality can be assessed independently of the reader, which is precisely the assumption we challenge\.

Query\-focused summarization \(QFS\)generates summaries tailored to a user\-provided query rather than producing a generic synopsis\([10](https://arxiv.org/html/2608.14457#bib.bib15);[9](https://arxiv.org/html/2608.14457#bib.bib16)\), with approaches ranging from early extractive methods that scored sentences by query relevance\([58](https://arxiv.org/html/2608.14457#bib.bib17);[44](https://arxiv.org/html/2608.14457#bib.bib18)\)to neural systems that generate query\-conditioned abstractive summaries\([3](https://arxiv.org/html/2608.14457#bib.bib19);[62](https://arxiv.org/html/2608.14457#bib.bib20);[55](https://arxiv.org/html/2608.14457#bib.bib21);[17](https://arxiv.org/html/2608.14457#bib.bib61)\), supported by datasets such as QMSum\([66](https://arxiv.org/html/2608.14457#bib.bib22)\)and AQuaMuSe\([33](https://arxiv.org/html/2608.14457#bib.bib23)\)\. A parallel line of work conditions summarization on reader attributes, including controllable systems that specify length, style, or focus\([14](https://arxiv.org/html/2608.14457#bib.bib24);[24](https://arxiv.org/html/2608.14457#bib.bib25)\), audience\-adaptive approaches targeting populations such as children, experts, or laypeople\([6](https://arxiv.org/html/2608.14457#bib.bib26);[1](https://arxiv.org/html/2608.14457#bib.bib27);[20](https://arxiv.org/html/2608.14457#bib.bib28);[41](https://arxiv.org/html/2608.14457#bib.bib29)\), and persona\-based prompting of large language models\([12](https://arxiv.org/html/2608.14457#bib.bib30);[26](https://arxiv.org/html/2608.14457#bib.bib31)\)\. However, QFS evaluation has largely relied on reference\-based metrics applied against query\-specific summaries\([66](https://arxiv.org/html/2608.14457#bib.bib22);[55](https://arxiv.org/html/2608.14457#bib.bib21)\); our work extends this framing by pairing each query with an explicit reader persona and interrogating whether existing metrics can detect query\-conditioned differences in informational content\.

Human EvaluationA number of datasets have been released with human annotations of summary quality, enabling meta\-evaluation of automatic metrics\. SummEval\([13](https://arxiv.org/html/2608.14457#bib.bib32)\)provides expert ratings of system outputs on CNN/DailyMail across coherence, consistency, fluency, and relevance\. FRANK\([45](https://arxiv.org/html/2608.14457#bib.bib33)\)contributes fine\-grained factual error annotations, and[53](https://arxiv.org/html/2608.14457#bib.bib34)extends this with error annotations across additional summarizers and datasets\. More recent benchmarks elicit preference judgments via pairwise comparison of system outputs\([21](https://arxiv.org/html/2608.14457#bib.bib35);[39](https://arxiv.org/html/2608.14457#bib.bib36)\)\. Across these resources, annotations are collected without reference to a particular reader’s goals; our evaluation anchors judgments to a specified persona and query, enabling direct measurement of information satisfaction\.

## 3\.Experimental Setup[RQ1](https://arxiv.org/html/2608.14457#S1.SS0.SSS0.Px1)

We evaluate the robustness of automatic summarization metrics by applying controlled perturbations to reference summaries and measuring whether each metric responds in the expected direction\.

We draw samples from 4 scientific summarization corpora that span a range of domains, document lengths, and intended audiences\.arXivandPubMed\([8](https://arxiv.org/html/2608.14457#bib.bib55)\)provide long\-document scientific articles paired with their abstracts, whileSciTLDR\([5](https://arxiv.org/html/2608.14457#bib.bib56)\)contributes extreme, single\-sentence TL;DR summaries of Computer Science papers\.eLifecontains editor\-written lay summaries of biomedical articles\([20](https://arxiv.org/html/2608.14457#bib.bib28)\)\. For each dataset, we use the test set and subsampleN=50N\{=\}50documents for experimentation\.

We evaluate a broad set of metrics covering the major families used in the summarization literature\. For lexical overlap with a reference summary, we include ROUGE\-1/2/L\([36](https://arxiv.org/html/2608.14457#bib.bib1)\), BLEU\([46](https://arxiv.org/html/2608.14457#bib.bib57)\), chrF\([47](https://arxiv.org/html/2608.14457#bib.bib37)\), and METEOR\([2](https://arxiv.org/html/2608.14457#bib.bib38)\)\. We additionally report a set of surface\-level statistics of the summary itself, computed via DataStats\([23](https://arxiv.org/html/2608.14457#bib.bib42)\): extractive coverage, extractive density, compression ratio, the percentage of novel 1/2/3\-grams relative to the source, and summary length\. A spaCy\-based\([25](https://arxiv.org/html/2608.14457#bib.bib48)\)syntactic\-complexity metric, measured in both words and sentences, is included as a reference\-free baseline that captures only surface properties of the summary\. We use the SumEval package to compute the above metrics\([13](https://arxiv.org/html/2608.14457#bib.bib32)\)\.

For non\-lexical, embedding\- and model\-based evaluation, we include BERTScore\([63](https://arxiv.org/html/2608.14457#bib.bib4)\), which computes token\-level cosine similarity using contextual embeddings\.111We use the default RoBERTa\-large model for BERTScore\.We further include the reference\-free metrics SUPERT\([18](https://arxiv.org/html/2608.14457#bib.bib39)\), SummaQA\([49](https://arxiv.org/html/2608.14457#bib.bib40)\), and BLANC\([54](https://arxiv.org/html/2608.14457#bib.bib41)\), which score summaries against the source document only\.

For LLM\-based evaluation, we run each metric under two judges, a larger Llama\-3\.3\-70B model222meta\-llama/Llama\-3\.3\-70B\-Instructand a smaller Prometheus\-7B model333prometheus\-eval/prometheus\-7b\-v2\.0\([28](https://arxiv.org/html/2608.14457#bib.bib58);[22](https://arxiv.org/html/2608.14457#bib.bib47)\), in order to assess judge sensitivity\. We include FActScore\([43](https://arxiv.org/html/2608.14457#bib.bib43)\), which verifies atomic facts against the source\. Following[38](https://arxiv.org/html/2608.14457#bib.bib11), we use an LLM\-as\-judge metric\([65](https://arxiv.org/html/2608.14457#bib.bib14)\)that scores summaries along the dimensions of relevance, consistency, fluency, coherence, and informativeness, and that also produces an overall score\. Finally, we include LLM\-as\-judge prompts that mirror the instructions provided to the human annotators \(*LLM Judge: Annotator*in our results below\)\.

Finally, we experiment with a variant of FActScore that measures the utility of information*nuggets*\([56](https://arxiv.org/html/2608.14457#bib.bib62);[57](https://arxiv.org/html/2608.14457#bib.bib63)\)to the user, rather than the fidelity of the nuggets to the source\. We refer to these metrics as*Persona Recall*and*Persona Precision*\. Similar to past work in claim decomposition\([60](https://arxiv.org/html/2608.14457#bib.bib64);[27](https://arxiv.org/html/2608.14457#bib.bib65)\), both decompose a summary into subclaims, via an LLM extraction step, then judge each nugget against the target persona \(role, domain, info needs, query\)\. Persona Precision measures the fraction of summary nuggets that an LLM judge deems relevant to the persona, penalizing off\-topic or persona\-irrelevant content\. Persona Recall first prompts the LLM to generate a list of information requirements implied by the persona and source document, then checks what fraction of those requirements is covered by the summary’s nuggets, penalizing omissions of persona\-critical information\. Together these metrics capture whether a summary contains the right information for a specific reader and nothing extraneous\.

### 3\.1\.Perturbation tests

We apply five perturbations to each summary, each producing an ordered sequence of progressively perturbed variants\. For each test, we state the*expected direction*in which a faithful metric should move as the perturbation level increases\.

Distractor sentences\(expected: decrease\)\. Up to five sentences sampled uniformly at random from*other*documents in the corpus are appended one at a time to the summary, following the style of distractor\-injection robustness probes\([31](https://arxiv.org/html/2608.14457#bib.bib44);[16](https://arxiv.org/html/2608.14457#bib.bib45)\)\.

Incremental addition\(expected: increase\)\. Starting from the empty string, we add one sentence of the original summary at a time until the full summary is reconstructed\. This probes whether a metric distinguishes partial from complete summaries\.

Lengthen\(expected: stable\)\. An LLM is prompted to expand the summary into longer prose without introducing new information; the source document is provided as grounding context\.

Shorten\(expected: stable\)\. The inverse of*lengthen*: the LLM is asked to condense the summary while preserving content\. Together these two tests measure length\-sensitivity, a known failure mode of overlap\-based metrics\([52](https://arxiv.org/html/2608.14457#bib.bib46)\)\.

Different audience\(expected: decrease\)\. The summary is rewritten by the LLM for a different target audience drawn from \{undergraduate student, journalist, domain expert\}\. This isolates the effect of audience\-aware emphasis, which persona\-aware metrics should detect as a quality drop relative to the original audience\.

We use Llama\-3\.3\-70B to generate the LLM\-based pertubations \(tests 3\-5\)\.

We compute the Spearman rank correlationρ\\rho\([51](https://arxiv.org/html/2608.14457#bib.bib49)\)between perturbation level and mean metric score\. We additionally report*Monotonicity*\(M\), the fraction of consecutive transitions between perturbation levels whose sign matches the expected direction\. For*stable*tests, a transition is considered correct when the absolute score change is below 5% of the baseline score\. The expected output of each statistic, along with each expected direction, is summarized in[Table 1](https://arxiv.org/html/2608.14457#S3.T1)\.

\\diaghead\(\-3,1\)\{\\hskip 62\.09747pt\}\{\{\\footnotesize\\shortstack\[l\]\{\{Stat\.\}\}\}\}\{\{\\footnotesize\\shortstack\[r\]\{\{Dir\.\}\}\}\}DecreaseIncreaseStableρ\\rho≈−1\\approx\-1≈\+1\\approx\+1≈0\\approx 0\(strong negative\)\(strong positive\)M≈1\.0\\approx 1\.0≈1\.0\\approx 1\.0≈1\.0\\approx 1\.0\(every step lowers the score\)\(every step raises the score\)\(every step stays within±5%\\pm 5\\%of baseline\)Table 1\.We report the expected statistic values \(Stat\.\) by the direction of change \(Dir\.\) for Spearman rank correlation \(ρ\\rho\) and Monotonicity \(M\)\. See §[3\.1](https://arxiv.org/html/2608.14457#S3.SS1)for details\.

## 4\.Human Evaluation Collection[RQ2](https://arxiv.org/html/2608.14457#S1.SS0.SSS0.Px2)

We conduct a human evaluation to assess whether conditioning summary generation on a reader’s self\-reported background yields summaries that are preferred by that reader\. Crucially, we measure preference with respect to the specific annotator who provided the query and persona, rather than aggregating judgments across annotators\. We release the annotation platform and data to support future research\.444[https://github\.com/JHU\-CLSP/persona\-eval\-metric](https://github.com/JHU-CLSP/persona-eval-metric)

Annotation protocol\.Each annotator interacts with a custom web\-based platform \(see[Figure1](https://arxiv.org/html/2608.14457#S4.F1)for screenshots\)\. A session proceeds in four stages: \(i\) the annotator completes a short pre\-task questionnaire eliciting self\-reported expertise, research area, and use case, which together constitute their*persona profile*; \(ii\) they submit a free\-form natural\-language query reflecting a genuine information need from their own work; \(iii\) the system retrieves the top 10 paper abstracts matching the query from OpenAlex555[https://openalex\.org/](https://openalex.org/)and conditions generation on this context; and \(iv\) the annotator ranks the resulting four summaries through a pairwise tournament \(described below\)\. Each annotator completes this loop for up to eight distinct queries\.

Summary generation\.For each query, four candidate summaries are produced from a2×22\\times 2design crossing two open\-weight instruction\-tuned models—Llama\-3\.3\-70B\([22](https://arxiv.org/html/2608.14457#bib.bib47)\)and DeepSeek\-V3\.1666deepseek\-ai/DeepSeek\-V3\.1\([11](https://arxiv.org/html/2608.14457#bib.bib54)\)—with two prompting conditions: one with the persona profile and one without\. All four summaries are generated from the same retrieved context \(to eliminate differences arising from search performance or stochasticity\) using a shared prompt template that instructs the model to ground its answer in the provided abstracts and to use bracketed in\-text citations\([35](https://arxiv.org/html/2608.14457#bib.bib50);[19](https://arxiv.org/html/2608.14457#bib.bib51)\)\.777Full prompts are provided in appendix\.

Ranking protocol\.Annotators rank the four summaries through a pairwise single\-elimination tournament: two pairs are compared in the first round, and the winners from each pair advance to a final\. Assignment of summaries to bracket positions is randomized so that annotators cannot infer which model or prompt condition produced a given summary\. Pairwise comparison has been shown to yield more reliable human judgments than absolute Likert ratings\([29](https://arxiv.org/html/2608.14457#bib.bib52);[40](https://arxiv.org/html/2608.14457#bib.bib53)\), and the bracket structure reduces annotator effort from\(42\)=6\\binom\{4\}\{2\}=6pairwise judgments to three while producing a general ordering over the top two summaries\. For each round, the annotators also have the option to select “neither” if neither summary meets their information needs\. If “neither” is selected for one or both of the first round pairs, the final round is skipped\.

Annotator recruitment\.Standard inter\-annotator agreement is not a meaningful quality signal in our setting; information satisfaction is defined relative to a*specific*reader’s background and query, and two annotators with different backgrounds should be expected to disagree on the same summary\. Agreement among annotators would in fact suggest that our protocol*fails*to capture the persona\-conditioned signal we aim to measure\. We instead ensure annotation quality through careful recruitment\. Rather than crowdsourcing, which would require annotators to fabricate information needs, we recruit 20 volunteer annotators directly through professional networks, all of whom hold genuine interests in supporting language technology research\. Our annotator pool spans domains including computer science, medicine, and materials science, and roles including researchers, practicing professionals \(e\.g\., nurses\), and undergraduate students\. Annotators were*not*required to use all assigned queries if they did not have a genuine information need for all eight\. Because summaries are generated live in response to each annotator’s own query, every judgment reflects an authentic information need\.

![Refer to caption](https://arxiv.org/html/2608.14457v1/figures/interface-intro.png)\(a\)Introduction to the task and instructions\.
![Refer to caption](https://arxiv.org/html/2608.14457v1/figures/interface-profile.png)\(b\)Profile questionnaire for building a persona\.
![Refer to caption](https://arxiv.org/html/2608.14457v1/figures/interface-query.png)\(c\)Annotators choose query topic\.
![Refer to caption](https://arxiv.org/html/2608.14457v1/figures/interface-comparison.png)\(d\)Example summary comparison\.

Figure 1\.Annotation workflow example\. Steps[1\(c\)](https://arxiv.org/html/2608.14457#S4.F1.sf3)\-[1\(d\)](https://arxiv.org/html/2608.14457#S4.F1.sf4)continue for each query assigned to the annotator\.DirectionalStabilityMetricDiffAud↓\\downarrowDistract↓\\downarrowIncrem↑\\uparrowLengthen≈\\approx0Shorten≈\\approx0LexicalROUGE\-1 \(F1\)−0\.40\-0\.40−1\.00\-1\.00−0\.63\-0\.63−1\.00\-1\.00−1\.00\-1\.00ROUGE\-2 \(F1\)−0\.40\-0\.40−0\.66\-0\.66−0\.63\-0\.63−1\.00\-1\.00−1\.00\-1\.00ROUGE\-L \(F1\)0\.200\.20−1\.00\-1\.00−0\.63\-0\.63−1\.00\-1\.00−1\.00\-1\.00BLEU−0\.20\-0\.20−1\.00\-1\.00−0\.64\-0\.64−1\.00\-1\.00−1\.00\-1\.00chrF−0\.40\-0\.40−1\.00\-1\.00−0\.60\-0\.60−1\.00\-1\.00−1\.00\-1\.00Extr\. Coverage−0\.40\-0\.40−0\.54\-0\.540\.020\.02−1\.00\-1\.00−1\.00\-1\.00Extr\. Density−0\.40\-0\.40−0\.54\-0\.540\.540\.54−1\.00\-1\.00−1\.00\-1\.00Compression Ratio−0\.40\-0\.40−1\.00\-1\.00−0\.65\-0\.65−1\.00\-1\.001\.001\.00Novel 1\-grams \(%\)0\.400\.401\.001\.000\.840\.841\.001\.001\.001\.00Novel 2\-grams \(%\)0\.400\.400\.830\.83−0\.09\-0\.091\.001\.001\.001\.00Novel 3\-grams \(%\)0\.400\.400\.540\.54−0\.40\-0\.401\.001\.001\.001\.00Summary Length0\.200\.201\.001\.001\.001\.001\.001\.00−1\.00\-1\.00METEOR−0\.40\-0\.40−1\.00\-1\.00−0\.56\-0\.56−1\.00\-1\.00−1\.00\-1\.00Synt\. Complexity \(W\)0\.200\.201\.001\.001\.001\.001\.001\.00−1\.00\-1\.00Synt\. Complexity \(S\)0\.200\.201\.001\.001\.001\.001\.001\.00−1\.00\-1\.00Non\-lexicalBERTScore \(F1\)−0\.40\-0\.40——−1\.00\-1\.00−1\.00\-1\.00BLANC0\.800\.80—−0\.31\-0\.311\.001\.001\.001\.00SummaQA \(F1\)———0\.000\.000\.000\.00SUPERT0\.500\.50————LLM judge: Llama\-70BFActScore0\.400\.40−0\.60\-0\.600\.380\.38−1\.00\-1\.00−1\.00\-1\.00Relevance—−0\.10\-0\.100\.420\.420\.000\.000\.000\.00Coherence0\.770\.77−0\.58\-0\.580\.580\.581\.001\.001\.001\.00Consistency—−0\.13\-0\.130\.420\.420\.000\.000\.000\.00Fluency0\.770\.770\.650\.650\.580\.581\.001\.001\.001\.00Informativeness0\.320\.320\.850\.850\.420\.421\.001\.001\.001\.00Overall0\.320\.32−0\.14\-0\.140\.580\.581\.001\.001\.001\.00Annotator−0\.77\-0\.77———−1\.00\-1\.00Persona Precision———0\.000\.000\.000\.00Persona Recall0\.800\.800\.200\.200\.450\.45−1\.00\-1\.00−1\.00\-1\.00LLM judge: Prometheus\-7BFActScore1\.001\.00−0\.40\-0\.40−0\.36\-0\.36−1\.00\-1\.00−1\.00\-1\.00Relevance−0\.21\-0\.21−0\.56\-0\.56−0\.20\-0\.20−1\.00\-1\.00−1\.00\-1\.00Coherence0\.400\.40−0\.12\-0\.12−0\.01\-0\.011\.001\.001\.001\.00Consistency−0\.40\-0\.40−0\.32\-0\.32−0\.17\-0\.17−1\.00\-1\.000\.000\.00Fluency0\.800\.80−0\.72\-0\.720\.200\.201\.001\.001\.001\.00Informativeness0\.800\.800\.750\.75−0\.11\-0\.111\.001\.001\.001\.00Overall0\.800\.80−0\.37\-0\.37−0\.04\-0\.041\.001\.001\.001\.00Annotator0\.400\.40−0\.65\-0\.65−0\.00\-0\.001\.001\.001\.001\.00Persona Precision0\.000\.000\.030\.03−0\.01\-0\.01−1\.00\-1\.00−1\.00\-1\.00Persona Recall−0\.60\-0\.60−0\.60\-0\.600\.090\.091\.001\.00−1\.00\-1\.00\(a\)Spearman’sρ\\rho\. Green when sign matches expected direction with\|ρ\|≥0\.4\|\\rho\|\\geq 0\.4\(or\|ρ\|≤0\.2\|\\rho\|\\leq 0\.2for stability tests\)\.
DirectionalStabilityMetricDiffAudDistractIncremLengthenShortenLexicalROUGE\-1 \(F1\)0\.671\.000\.270\.000\.00ROUGE\-2 \(F1\)0\.670\.800\.200\.000\.00ROUGE\-L \(F1\)0\.671\.000\.200\.000\.00BLEU0\.331\.000\.130\.000\.00chrF0\.671\.000\.200\.001\.00Extr\. Coverage0\.670\.800\.530\.000\.00Extr\. Density0\.670\.800\.530\.000\.00Compression Ratio0\.671\.000\.070\.000\.00Novel 1\-grams \(%\)0\.330\.000\.710\.000\.00Novel 2\-grams \(%\)0\.330\.200\.570\.000\.00Novel 3\-grams \(%\)0\.330\.200\.570\.000\.00Summary Length0\.670\.001\.000\.000\.00METEOR0\.671\.000\.270\.000\.00Synt\. Complexity \(W\)0\.670\.001\.000\.000\.00Synt\. Complexity \(S\)0\.670\.001\.000\.000\.00Non\-lexicalBERTScore \(F1\)0\.671\.000\.000\.000\.00BLANC0\.330\.000\.071\.001\.00SummaQA \(F1\)0\.000\.000\.001\.001\.00SUPERT0\.50——1\.001\.00LLM judge: Llama\-70BFActScore0\.330\.600\.470\.000\.00Relevance0\.000\.400\.071\.001\.00Coherence0\.000\.400\.130\.001\.00Consistency0\.000\.200\.071\.001\.00Fluency0\.000\.000\.130\.000\.00Informativeness0\.330\.000\.070\.000\.00Overall0\.330\.800\.130\.001\.00Annotator0\.330\.00—1\.000\.00Persona Precision0\.000\.000\.001\.001\.00Persona Recall0\.330\.600\.600\.000\.00LLM judge: Prometheus\-7BFActScore0\.000\.750\.380\.000\.00Relevance0\.670\.400\.271\.000\.00Coherence0\.330\.400\.530\.000\.00Consistency0\.670\.400\.270\.001\.00Fluency0\.330\.600\.470\.000\.00Informativeness0\.330\.200\.400\.000\.00Overall0\.330\.600\.470\.000\.00Annotator0\.330\.200\.270\.000\.00Persona Precision0\.670\.800\.431\.000\.00Persona Recall0\.330\.600\.310\.000\.00\(b\)Match\-rateMM\(fraction of instances where metric responds as expected\)\. Green whenM≥0\.5M\\geq 0\.5\.

Table 2\.Spearman’sρ\\rho\(left\) and match\-rateMM\(right\) across all perturbation tests\.
## 5\.Perturbation Testing Results[RQ1](https://arxiv.org/html/2608.14457#S1.SS0.SSS0.Px1)

Results for the five perturbation tests for all metrics described in[Section3](https://arxiv.org/html/2608.14457#S3)are presented in[Table2](https://arxiv.org/html/2608.14457#S4.T2)\. For the directional tests, we consider the test “passed” if\|ρ\|≥0\.4\|\{\\rho\}\|\\geq 0\.4in the expected direction; in other words, if the test produces at least a moderate correlation\. For the stability tests, we consider the test “passed” if\|ρ\|≤0\.2\|\{\\rho\}\|\\leq 0\.2, or if the test produces a weak correlation\. For monotonicity, we consider the test “passed” ifM≥0\.5M\\geq 0\.5, or if the metric moves in the expected direction at least half the time\. We release the code for the metric calculation, perturbation testing, and analysis to support future research\.888[https://github\.com/JHU\-CLSP/persona\-eval\-annotation](https://github.com/JHU-CLSP/persona-eval-annotation)

Distractor sentences\.Appending unrelated sentences should cause reference\-based metrics to decrease\. The lexical overlap metrics \(ROUGE\-1/L, BLEU, chrF, METEOR, compression\) behave as expected, all reachingρ=−1\.0\\rho=\-1\.0withM=1\.0M=1\.0\. More notably, BERTScore, BLANC, and SummaQA fail to produce a defined correlation, and the LLM\-based metrics are inconsistent across judges: under Llama\-3\.3\-70B only FActScore \(ρ=−0\.60\\rho=\-0\.60\) and coherence \(ρ=−0\.58\\rho=\-0\.58\) pass, while under Prometheus\-7B a different and largely non\-overlapping subset passes \(relevance, fluency, the annotator\-style judge, persona recall\)\. The metrics most often advocated as “content\-aware” alternatives to ROUGE are thus the*least*reliable detectors of obviously off\-topic content\.

Incremental addition\.Reconstructing the summary one sentence at a time should cause scores to increase as the summary becomes more complete\. Strikingly, ROUGE\-1/2/L, BLEU, chrF, METEOR, and compression all show*negative*correlations between−0\.56\-0\.56and−0\.65\-0\.65\. Partial summaries are scored*higher*than complete ones, likely an artifact of length normalization rewarding short prefixes that overlap the reference\. The Llama\-3\.3\-70B judge is the lone bright spot: every prompt\-driven dimension lies betweenρ=0\.42\\rho=0\.42andρ=0\.58\\rho=0\.58in the correct direction, although low monotonicity \(M≤0\.13M\\leq 0\.13\) indicates noisy scores\. The Prometheus\-7B judge fails on every dimension\. No reference\-based metric in either the lexical or non\-lexical family reliably distinguishes partial from complete summaries\.

Lengthen and shorten prose\.A content\-preserving rewrite that changes surface length should leave a content\-aware metric approximately unchanged \(ρ≈0\\rho\\approx 0\)\. Almost every metric fails: all overlap\-based metrics, extractive coverage and density, compression, and BERTScore reachρ=−1\.0\\rho=\-1\.0onlengthen\_prose, with mirrored behavior onshorten\_prose\. Most LLM Judge dimensions also collapse to\|ρ\|=1\.0\|\\rho\|=1\.0, indicating they are themselves strongly length\-sensitive\. The few length\-invariant metrics are SummaQA, SUPERT, and a small subset of LLM\-based dimensions \(Llama\-3\.3\-70B relevance, consistency, and persona precision; Prometheus\-7B consistency\)\. This extends the length\-sensitivity failure mode previously documented for ROUGE\([52](https://arxiv.org/html/2608.14457#bib.bib46)\)to BERTScore and to most LLM\-judge dimensions\.

Different audience\.Rewriting a summary for a different target audience should be detected as a quality drop\. The most striking result is that several LLM\-based metrics move in the*wrong*direction: under Prometheus\-7B, fluency, informativeness, overall, and FActScore all show strong positive correlations of0\.800\.80–1\.001\.00, rating audience\-shifted summaries as*higher*quality than the original\. The lexical overlap metrics \(ROUGE\-1/2, chrF, METEOR, coverage, density, compression\) and BERTScore sit exactly at our pass threshold \(ρ=−0\.40\\rho=\-0\.40\), giving only a weak directional signal, and ROUGE\-L, BLEU, BLANC, SummaQA, and SUPERT fail outright\. Under the Llama\-3\.3\-70B judge, only the annotator\-style judge passes \(ρ=−0\.77\\rho=\-0\.77\); under Prometheus, only consistency \(ρ=−0\.40\\rho=\-0\.40\) and persona recall \(ρ=−0\.60\\rho=\-0\.60\)\. Compared to the much stronger responses these same metrics produce under distractor injection, this confirms that existing summarization metrics are far more sensitive to surface\-level corruptions than to meaningful changes in the audience the summary serves\.

Summary of robustness findings\.No single metric passes all of the tests it should, and most pass only the distractor\-injection test—the perturbation furthest from realistic variation\. LLM\-based metrics offer a partial complement, recovering sensitivity on incremental addition \(where lexical metrics catastrophically fail\), but they are highly inconsistent across judge models: dimensions that pass under Llama\-3\.3\-70B routinely fail or invert under Prometheus\-7B\. Most importantly, every evaluated metric is approximately insensitive to, or even*anticorrelated*with, audience\-shift rewrites—the dimension along which informational satisfaction varies between users\. These results motivate the human evaluation in[Section6](https://arxiv.org/html/2608.14457#S6)\.

Metricα\\alphaROUGE\-1 F10\.077ROUGE\-2 F10\.117ROUGE\-L F10\.032chrF0\.069SummaQA F10\.173BERTScore F1\-0\.029BLEU0\.024METEOR0\.018BLANC0\.018SUPERT\-0\.005\(a\)Traditional metrics\.
Metricα\\alphaOverall\-0\.036Relevance0\.001Coherence0\.158Consistency0\.027Fluency\-0\.122Informativeness0\.083FactScore\-0\.009Persona\-precision\-0\.150Persona\-recall\-0\.150\(b\)LLM\-as\-a\-judge \(Prometheus\-7B\)\.
Metricα\\alphaOverall\-0\.103Relevance\-0\.153Coherence\-0\.106Consistency\-0\.102Fluency0\.000Informativeness\-0\.179FactScore0\.053Persona\-precision\-0\.105Persona\-recall\-0\.087\(c\)LLM\-as\-a\-judge \(Llama\-3\.3\-70B\-Instruct\)\.

Table 3\.α\\alphabetween metrics and human pairwise preferences\. Chance agreement corresponds toα=0\\alpha=0\. No metric family substantively exceeds chance, including persona\-aware variants designed to capture informational satisfaction\.
## 6\.Human Eval Results[RQ2](https://arxiv.org/html/2608.14457#S1.SS0.SSS0.Px2)

We collected 140 completed query sessions from 20 annotators, yielding 420 pairwise comparisons across the three\-match tournament described in[Section4](https://arxiv.org/html/2608.14457#S4)\. On average, annotators spent roughly four minutes per page \(median 258s\)\. Sixteen annotators completed all eight assigned queries\. For reference\-based metrics, we concatenate the titles of the retrieved papers as the reference\. For metrics that require the source, we concatenate the abstracts of the retrieved papers\.

DeepSeek\-V3\.1 is preferred over Llama\-3\.3\-70B\.Aggregating across both prompting conditions and all tournament rounds, DeepSeek\-V3\.1 wins 72\.8% of its matchups \(91/125\) in the final round, against 27\.2% for Llama\-3\.3\-70B\-Instruct\. The gap is even more pronounced in direct cross\-model first\-round matchups, where DeepSeek wins 73\.1% of head\-to\-head comparisons \(68/93\)\. This preference is consistent across the four\-variant breakdown shown in[Table4\(a\)](https://arxiv.org/html/2608.14457#S6.T4.st1)\. Both DeepSeek variants outperform both Llama variants in both rounds\.

Personalization is preferred when generated by a stronger model\.The effect of conditioning on the persona profile depends on where in the tournament the comparison occurs \([Table4\(b\)](https://arxiv.org/html/2608.14457#S6.T4.st2)\)\. In the final round, annotators prefer the personalized summary 63\.2% of the time \(79/125\) over the generic summary, with 15 ties \(“neither”\)\. The first\-round picture is less clean: in the*A vs\. B*bracket, personalized summaries win only 46\.5% of the time \(60/129\) while in the*C vs\. D*bracket, personalized summaries win 62\.2% \(79/127\), matching the final\-round rate\. We hypothesize this effect is a result of the fact that the first\-round result aggregates across both models, and our cross\-model comparisons are dominated by a strong preference for DeepSeek\-V3\.1 over Llama\-3\.3\-70B \([Table4\(a\)](https://arxiv.org/html/2608.14457#S6.T4.st1)\)\. If Llama is simply less effective at personalization, then including its personalized outputs in the first\-round average would dilute the personalization signal there, while the final round, which is disproportionately populated by the stronger model’s outputs, would surface it more cleanly\.

Round 1FinalVariantW/N%ppW/N%ppDeepSeek\-V3\.1 \(pers\.\)88/14062\.9<<0\.00158/8865\.90\.002DeepSeek\-V3\.1 \(gen\.\)75/14053\.60\.01033/7544\.00\.474Llama\-3\.3\-70B \(pers\.\)51/14036\.40\.06621/5141\.20\.284Llama\-3\.3\-70B \(gen\.\)42/14030\.0<<0\.00113/4231\.00\.112\(a\)Win rates for each of the four summary variants in round 1 \(out of 140 round\-1 matchups per variant\) and in the final round \(out of the variant’s round\-1 wins\)\. We compare each model for both the generic \(gen\.\) and personalized \(pers\.\) settings\. DeepSeek\-V3\.1 with persona conditioning is the most preferred variant across both rounds\. Theppcolumns report annotator\-clustered bootstrap significance against a 50/50 null \(two\-sided, decided matchups only\)\.
ComparisonPers\.Gen\.TieppRound 1 \(A vs\. B\)60 \(46\.5%\)69 \(53\.5%\)110\.408Round 1 \(C vs\. D\)79 \(62\.2%\)48 \(37\.8%\)130\.015Final round79 \(63\.2%\)46 \(36\.8%\)150\.026Same\-model R151 \(57\.3%\)38 \(42\.7%\)—0\.239\(b\)Annotator preferences between persona\-conditioned and generic summaries, by tournament position\. Personalization is preferred in the final round and in the second first\-round bracket \(C vs\. D\), and—when model identity is held fixed—in same\-model first\-round matchups\. Theppcolumn reports annotator\-clustered bootstrap significance against a 50/50 null; bold indicatesp<0\.05p<0\.05\.
TermCoef\.SEzzppOR \[95% CI\]Intercept0\.6190\.3111\.990\.0461\.86 \[1\.01, 3\.42\]model\_pq= Llama\-3\.3\-70B\-0\.8330\.347\-2\.400\.0160\.43 \[0\.22, 0\.86\]model\_generic= Llama\-3\.3\-70B0\.5230\.2951\.770\.0761\.69 \[0\.95, 3\.01\]round\_stage= round 1\-0\.1990\.319\-0\.620\.5330\.82 \[0\.44, 1\.53\]\(c\)Logistic regression of personalized\-side win on personalized\-side model, generic\-side model, and tournament round\. Reference levels:model\_pq= DeepSeek\-V3\.1,model\_generic= DeepSeek\-V3\.1,round\_stage= final\. Standard errors are cluster\-robust by annotator\.N=248N=248comparisons; personalized side won 58\.5% of comparisons overall\.

Table 4\.Human evaluation results\. \([4\(a\)](https://arxiv.org/html/2608.14457#S6.T4.st1)\) Per\-variant win rates across the2×22\\times 2design\. \([4\(b\)](https://arxiv.org/html/2608.14457#S6.T4.st2)\) Aggregated personalized\-vs\-generic preferences by tournament position\. \([4\(c\)](https://arxiv.org/html/2608.14457#S6.T4.st3)\) Logistic regression of the personalized\-side win with cluster\-robust standard errors\.To test this hypothesis while controlling for model strength, we restricted the data to comparisons in which one summary was personalized and the other generic, excluding same\-condition matchups and “neither” picks\. Each remaining comparison \(N = 248, from 19 annotators\) contributed one observation with a binary outcome indicating whether the personalized side won\. We then fit a logistic regression ofpersonalized\_wonon three categorical predictors: the model on the personalized side \(model\_pq\), the model on the generic side \(model\_generic\), and the round in which the comparison occurred \(round\_stage: round 1 or final\), with DeepSeek\-V3\.1 and the final round as reference levels\. Because each annotator contributed multiple comparisons, we computed cluster\-robust standard errors grouped by annotator\([42](https://arxiv.org/html/2608.14457#bib.bib60)\)\. Under this parameterization, the intercept estimates the log\-odds of the personalized variant winning at the reference levels \(personalized DeepSeek vs\. generic DeepSeek in the final round\), and thus, after exponentiation to an odds ratio, directly quantifies the personalization effect\.

We report the results of the regression analysis in[Table4\(c\)](https://arxiv.org/html/2608.14457#S6.T4.st3), which support the model\-strength hypothesis\. The personalized DeepSeek summary is preferred over the generic DeepSeek summary with an odds ratio of1\.861\.86\(95% CI\[1\.01,3\.42\]\[1\.01,3\.42\],p=0\.046p=0\.046\)\. The penalty for putting Llama on the personalized side is large and significant:OR=0\.43\\text\{OR\}=0\.43\(\[0\.22,0\.86\]\[0\.22,0\.86\],p=0\.016p=0\.016\), meaning that swapping the personalized model from DeepSeek to Llama more than halves the odds that the personalized side wins\. The coefficient on the generic\-side model trends in the expected complementary direction \(OR=1\.69\\text\{OR\}=1\.69,\[0\.95,3\.01\]\[0\.95,3\.01\],p=0\.076p=0\.076\): personalization wins more often when the generic opponent is Llama, though this effect is not significant atα=0\.05\\alpha=0\.05\. The round\-stage coefficient is small and non\-significant \(OR=0\.82\\text\{OR\}=0\.82,\[0\.44,1\.53\]\[0\.44,1\.53\],p=0\.53p=0\.53\), indicating that once model identity is controlled for, the apparent round\-level differences in raw win rates largely vanish\. This is consistent with the claim that those differences were driven by the composition of models in each round rather than by the round itself\.

Automatic and LLM\-based metrics correlate poorly with human judgment\.To assess whether existing summarization metrics recover the human preferences just described, we compute Krippendorff’sα\\alphabetween each metric and the annotator’s choice on every comparison in our evaluation\([30](https://arxiv.org/html/2608.14457#bib.bib59)\)\. A metric agrees with the annotator on a given pair if it assigns a higher score to the summary that the annotator selected, corresponding toα≈0\\alpha\\approx 0\. We count the metric’s decision as “neither” if the scores are within a per\-metric threshold of similarity, reported in the appendix\.

[Table3](https://arxiv.org/html/2608.14457#S5.T3)reports Krippendorff’sα\\alphagrouped into three families: reference\-based and reference\-free automatic metrics \(e\.g\., ROUGE, BERTScore, BLANC\), LLM\-as\-a\-judge scores using Prometheus\-7B as the evaluator, and LLM\-as\-a\-judge scores using Llama\-3\.3\-70B\-Instruct as the evaluator\. Across all three families, chance\-corrected agreement remains at or near chance \(α=0\\alpha=0\): automatic metrics range from−0\.029\-0\.029to0\.1730\.173\(mean0\.0490\.049\), Prometheus\-based judgments range from−0\.150\-0\.150to0\.1580\.158\(mean−0\.022\-0\.022\), and Llama\-based judgments range from−0\.179\-0\.179to0\.0530\.053\(mean−0\.087\-0\.087\), with all but FactScore \(0\.0530\.053\) and Fluency \(0\.0000\.000\) falling below chance\. Notably, the persona\-aware variants we designed specifically to capture informational satisfaction,persona\_precisionandpersona\_recall, are among the worst performers under the Prometheus judge \(α=−0\.150\\alpha=\-0\.150for both\) and remain at or below chance under Llama, withα\\alphavalues across the two judges ranging from−0\.150\-0\.150to−0\.087\-0\.087, indicating systematic disagreement with annotator preferences rather than mere noise\.

The strongest performers are reference\-based metrics anchored to a reference summary \(SummaQA F1 atα=0\.173\\alpha=0\.173, ROUGE\-2 F1 at0\.1170\.117, ROUGE\-1 F1 at0\.0770\.077\), but even these correspond to only slight agreement on conventional interpretations ofα\\alphaand fall far short of the level required to use them as proxies for personalized informational satisfaction\. The LLM\-judge protocols, including those explicitly conditioned on the annotator’s persona profile, perform no better than reference\-free automatic metrics\. Together, these results support the central claim of this paper: existing summarization metrics, including state\-of\-the\-art LLM\-based judges, are insufficient measures of how well a summary serves an individual reader’s informational needs\.

## 7\.Conclusion

We introducedinformation satisfactionas a user\-centered axis of summarization evaluation and asked whether existing metrics can measure it\. The answer is no: Our perturbation tests reveal that traditional metrics, embedding\-based metrics, and even strong LLM\-as\-judge protocols fail basic robustness checks\. Most notably, nearly every metric we evaluated is insensitive to \(or even*anticorrelated*with\) audience\-shift rewrites—the very dimension along which informational satisfaction varies between users\. Our expert human evaluation confirms the consequence, as no metric family exceeds chance agreement with reader preferences anchored to a specific query and persona\. Progress on user\-centered summarization will therefore require evaluation frameworks that explicitly model who the summary is for and what they seek to learn\. Conditioning on the persona is not by itself sufficient, however\. Our persona metric variants incorporate the reader’s role, domain, and information needs, yet do not consistently pass the perturbation or human correlation tests\. Their failure indicates that the bottleneck is not the absence of persona information but how satisfaction is measured\. Both metrics reduce satisfaction to the presence or absence of nuggets that an LLM judges relevant\. This assumes the model’s estimate of a reader’s needs matches the reader’s own, and it treats every information requirement as equally and independently satisfiable\. A metric that truly measures information satisfaction will instead need to weight information by its importance to the specific reader, account for what that reader already knows, and capture the comparative judgment a person makes when one adequate summary serves them better than another\. Our analysis and human\-annotated dataset provide a foundation for future research on evaluation of information satisfaction\.

## References

- Augustet al\.\(2022\)T\. August, L\. L\. Wang, J\. Bragg, M\. A\. Hearst, A\. Head, and K\. LoPaperPlain: making medical research papers approachable to healthcare consumers with natural language processing\.InACM Transactions on Computer\-Human Interaction,Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Banerjee and Lavie \(2005\)S\. Banerjee and A\. LavieMETEOR: an automatic metric for MT evaluation with improved correlation with human judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,pp\. 65–72\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Baumelet al\.\(2018\)T\. Baumel, M\. Eyal, and M\. ElhadadQuery focused abstractive summarization: incorporating query relevance, multi\-document coverage, and summary length constraints into seq2seq models\.External Links:1801\.07704Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Bhandariet al\.\(2020\)M\. Bhandari, P\. N\. Gour, A\. Ashfaq, P\. Liu, and G\. NeubigRe\-evaluating evaluation in text summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9347–9359\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Cacholaet al\.\(2020\)I\. Cachola, K\. Lo, A\. Cohan, and D\. S\. WeldTLDR: extreme summarization of scientific documents\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4766–4777\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p2.1)\.
- Chandrasekaranet al\.\(2020\)M\. K\. Chandrasekaran, G\. Feigenblat, E\. Hovy, A\. Ravichander, M\. Shmueli\-Scheuer, and A\. de WaardOverview and insights from the shared tasks at scholarly document processing 2020: CL\-SciSumm, LaySumm and LongSumm\.InProceedings of the First Workshop on Scholarly Document Processing,pp\. 214–224\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Chiang and Lee \(2023\)C\. Chiang and H\. LeeCan large language models be an alternative to human evaluations?\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 15607–15631\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Cohanet al\.\(2018\)A\. Cohan, F\. Dernoncourt, D\. S\. Kim, T\. Bui, S\. Kim, W\. Chang, and N\. GoharianA discourse\-aware attention model for abstractive summarization of long documents\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 615–621\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p2.1)\.
- Dang \(2005\)H\. T\. DangOverview of DUC 2005\.InProceedings of the Document Understanding Conference \(DUC\),Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Daumé III and Marcu \(2006\)H\. Daumé III and D\. MarcuBayesian query\-focused summarization\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 305–312\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§4](https://arxiv.org/html/2608.14457#S4.p3.1)\.
- Deshpandeet al\.\(2023\)A\. Deshpande, V\. Murahari, T\. Rajpurohit, A\. Kalyan, and K\. NarasimhanToxicity in ChatGPT: analyzing persona\-assigned language models\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 1236–1270\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Fabbriet al\.\(2021\)A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. RadevSummEval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics \(TACL\)9,pp\. 391–409\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§2](https://arxiv.org/html/2608.14457#S2.p3.1),[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Fanet al\.\(2018\)A\. Fan, D\. Grangier, and M\. AuliControllable abstractive summarization\.InProceedings of the 2nd Workshop on Neural Machine Translation and Generation,pp\. 45–54\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Fuet al\.\(2024\)J\. Fu, S\. Ng, Z\. Jiang, and P\. LiuGPTScore: evaluate as you desire\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Gabrielet al\.\(2021\)S\. Gabriel, A\. Celikyilmaz, R\. Jha, Y\. Choi, and J\. GaoGO FIGURE: a meta evaluation of factuality in summarization\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 478–487\.Cited by:[§3\.1](https://arxiv.org/html/2608.14457#S3.SS1.p2.1)\.
- Ganttet al\.\(2024\)W\. Gantt, A\. Martin, P\. Kuchmiichuk, and A\. S\. WhiteEvent\-keyed summarization\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7333–7345\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.431/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.431)Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Gaoet al\.\(2020\)Y\. Gao, W\. Zhao, and S\. EgerSUPERT: towards new frontiers in unsupervised evaluation metrics for multi\-document summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1347–1354\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p4.1)\.
- Gaoet al\.\(2023\)Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. WangRetrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§4](https://arxiv.org/html/2608.14457#S4.p3.1)\.
- Goldsacket al\.\(2022\)T\. Goldsack, Z\. Zhang, C\. Lin, and C\. ScartonMaking science simple: corpora for the lay summarisation of scientific literature\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 10589–10604\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1),[§3](https://arxiv.org/html/2608.14457#S3.p2.1)\.
- Goyalet al\.\(2022\)T\. Goyal, J\. J\. Li, and G\. DurrettNews summarization and evaluation in the era of GPT\-3\.External Links:2209\.12356Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p3.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, and et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p5.1),[§4](https://arxiv.org/html/2608.14457#S4.p3.1)\.
- Gruskyet al\.\(2018\)M\. Grusky, M\. Naaman, and Y\. ArtziNewsroom: a dataset of 1\.3 million summaries with diverse extractive strategies\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 708–719\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Heet al\.\(2022\)J\. He, W\. Kryściński, B\. McCann, N\. Rajani, and C\. XiongCTRLsum: towards generic controllable text summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5879–5915\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Honnibalet al\.\(2020\)M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. BoydspaCy: industrial\-strength natural language processing in Python\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.1212303)Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Jianget al\.\(2024\)H\. Jiang, X\. Zhang, X\. Cao, C\. Breazeal, D\. Roy, and J\. KabbaraPersonaLLM: investigating the ability of large language models to express personality traits\.InFindings of the Association for Computational Linguistics: NAACL,Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Jianget al\.\(2025\)Z\. Jiang, J\. Zhang, N\. Weir, S\. Ebner, M\. Wanner, K\. Sanders, D\. Khashabi, A\. Liu, and B\. Van DurmeCore: robust factual precision with informative sub\-claim identification\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19833–19856\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1018/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1018),ISBN 979\-8\-89176\-256\-5Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p6.1)\.
- Kimet al\.\(2023\)S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. SeoPrometheus: inducing fine\-grained evaluation capability in language models\.External Links:2310\.08491Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p5.1)\.
- Kiritchenko and Mohammad \(2017\)S\. Kiritchenko and S\. M\. MohammadBest\-worst scaling more reliable than rating scales: a case study on sentiment intensity annotation\.Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(ACL\)\.Cited by:[§4](https://arxiv.org/html/2608.14457#S4.p4.1)\.
- Krippendorff \(2004\)K\. KrippendorffMeasuring the reliability of qualitative text analysis data\.Quality and Quantity38,pp\. 787–800\.External Links:[Link](https://api.semanticscholar.org/CorpusID:57316832)Cited by:[§6](https://arxiv.org/html/2608.14457#S6.p6.1)\.
- Kryścińskiet al\.\(2019\)W\. Kryściński, N\. S\. Keskar, B\. McCann, C\. Xiong, and R\. SocherNeural text summarization: a critical evaluation\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 540–551\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.14457#S3.SS1.p2.1)\.
- Kryścińskiet al\.\(2020\)W\. Kryściński, B\. McCann, C\. Xiong, and R\. SocherEvaluating the factual consistency of abstractive text summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9332–9346\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Kulkarniet al\.\(2020\)S\. Kulkarni, S\. Chammas, W\. Zhu, F\. Sha, and E\. IeAQuaMuSe: automatically generating datasets for query\-based multi\-document summarization\.External Links:2010\.12694Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Labanet al\.\(2022\)P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. HearstSummaC: re\-visiting NLI\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics \(TACL\)10,pp\. 163–177\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4](https://arxiv.org/html/2608.14457#S4.p3.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§1](https://arxiv.org/html/2608.14457#S1.p2.1),[§2](https://arxiv.org/html/2608.14457#S2.p1.1),[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Liu and Liu \(2008\)F\. Liu and Y\. LiuCorrelation between ROUGE and human evaluation of extractive meeting summaries\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 201–204\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2511–2522\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§1](https://arxiv.org/html/2608.14457#S1.p2.1),[§2](https://arxiv.org/html/2608.14457#S2.p1.1),[§3](https://arxiv.org/html/2608.14457#S3.p5.1)\.
- Liuet al\.\(2024\)Y\. Liu, A\. R\. Fabbri, J\. Chen, Y\. Zhao, S\. Han, S\. Joty, P\. Liu, D\. Radev, C\. Wu, and A\. CohanBenchmarking generation and evaluation capabilities of large language models for instruction controllable summarization\.InFindings of the Association for Computational Linguistics: NAACL,Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p3.1)\.
- Liusieet al\.\(2024\)A\. Liusie, P\. Manakul, and M\. J\. F\. GalesLLM comparative assessment: zero\-shot NLG evaluation through pairwise comparisons using large language models\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),Cited by:[§4](https://arxiv.org/html/2608.14457#S4.p4.1)\.
- Luoet al\.\(2022\)Z\. Luo, Q\. Xie, and S\. AnaniadouReadability controllable biomedical document summarization\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 4667–4680\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- MacKinnonet al\.\(2023\)J\. G\. MacKinnon, M\. Ø\. Nielsen, and M\. D\. WebbCluster\-robust inference: a guide to empirical practice\.Journal of Econometrics232\(2\),pp\. 272–299\.External Links:ISSN 0304\-4076,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.jeconom.2022.04.001),[Link](https://www.sciencedirect.com/science/article/pii/S0304407622000781)Cited by:[§6](https://arxiv.org/html/2608.14457#S6.p4.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 12076–12100\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p5.1)\.
- Otterbacheret al\.\(2009\)J\. Otterbacher, G\. Erkan, and D\. R\. RadevBiased LexRank: passage retrieval using random walks with question\-based priors\.InInformation Processing and Management,Vol\.45,pp\. 42–54\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Pagnoniet al\.\(2021\)A\. Pagnoni, V\. Balachandran, and Y\. TsvetkovUnderstanding factuality in abstractive summarization with FRANK: a benchmark for factuality metrics\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 4812–4829\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p3.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 311–318\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Popović \(2017\)M\. PopovićchrF\+\+: words helping character n\-grams\.InProceedings of the Second Conference on Machine Translation \(WMT\),pp\. 612–618\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p3.1)\.
- Scialomet al\.\(2021\)T\. Scialom, P\. Dray, S\. Lamprier, B\. Piwowarski, J\. Staiano, A\. Wang, and P\. GallinariQuestEval: summarization asks for fact\-based evaluation\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6594–6604\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Scialomet al\.\(2019\)T\. Scialom, S\. Lamprier, B\. Piwowarski, and J\. StaianoAnswers unite\! Unsupervised metrics for reinforced summarization models\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 3246–3256\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p4.1)\.
- Sellamet al\.\(2020\)T\. Sellam, D\. Das, and A\. P\. ParikhBLEURT: learning robust metrics for text generation\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 7881–7892\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Spearman \(1904\)C\. SpearmanThe proof and measurement of association between two things\.The American Journal of Psychology15\(1\),pp\. 72–101\.Cited by:[§3\.1](https://arxiv.org/html/2608.14457#S3.SS1.p8.1)\.
- Sunet al\.\(2019\)S\. Sun, O\. Shapira, I\. Dagan, and A\. NenkovaHow to compare summarizers without target length? Pitfalls, solutions and re\-examination of the neural summarization literature\.InProceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation \(NeuralGen\),pp\. 21–29\.Cited by:[§3\.1](https://arxiv.org/html/2608.14457#S3.SS1.p5.1),[§5](https://arxiv.org/html/2608.14457#S5.p4.1)\.
- Tanget al\.\(2022\)L\. Tang, T\. Goyal, A\. R\. Fabbri, P\. Laban, J\. Xu, S\. Yavuz, W\. Kryscinski, J\. F\. Rousseau, and G\. DurrettUnderstanding factual errors in summarization: errors, summarizers, datasets, error detectors\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p3.1)\.
- Vasilyevet al\.\(2020\)O\. Vasilyev, V\. Dharnidharka, and J\. BohannonFill in the BLANC: human\-free quality estimation of document summaries\.InProceedings of the First Workshop on Evaluation and Comparison of NLP Systems,pp\. 11–20\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p4.1)\.
- Viget al\.\(2022\)J\. Vig, A\. R\. Fabbri, W\. Kryscinski, C\. Wu, and W\. LiuExploring neural models for query\-focused summarization\.InFindings of the Association for Computational Linguistics: NAACL,pp\. 1455–1468\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Voorhees \(2003a\)E\. M\. VoorheesEvaluating answers to definition questions\.InCompanion Volume of the Proceedings of HLT\-NAACL 2003 \- Short Papers,pp\. 109–111\.External Links:[Link](https://aclanthology.org/N03-2037/)Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p6.1)\.
- Voorhees \(2003b\)E\. M\. VoorheesOverview of the trec 2003 question answering track\.\.InTREC,Vol\.2003,pp\. 54–68\.Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p6.1)\.
- Wanet al\.\(2007\)X\. Wan, J\. Yang, and J\. XiaoManifold\-ranking based topic\-focused multi\-document summarization\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 2903–2908\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Wanget al\.\(2020\)A\. Wang, K\. Cho, and M\. LewisAsking and answering questions to evaluate the factual consistency of summaries\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 5008–5020\.Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Wanneret al\.\(2025\)M\. Wanner, B\. Van Durme, and M\. DredzeDnDScore: decontextualization and decomposition for factuality verification in long\-form text generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 23609–23626\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1205/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1205),ISBN 979\-8\-89176\-332\-6Cited by:[§3](https://arxiv.org/html/2608.14457#S3.p6.1)\.
- Wuet al\.\(2025\)M\. Wu, M\. M\. Hossain, T\. Wood, S\. A\. Akbar, S\. Chin, and E\. CornejoSEEval: advancing LLM text evaluation efficiency and accuracy through self\-explanation prompting\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 7372–7383\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.411/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.411),ISBN 979\-8\-89176\-195\-7Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p2.1)\.
- Xu and Lapata \(2020\)Y\. Xu and M\. LapataCoarse\-to\-fine query focused multi\-document summarization\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3632–3645\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with BERT\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§1](https://arxiv.org/html/2608.14457#S1.p2.1),[§2](https://arxiv.org/html/2608.14457#S2.p1.1),[§3](https://arxiv.org/html/2608.14457#S3.p4.1)\.
- Zhaoet al\.\(2019\)W\. Zhao, M\. Peyrard, F\. Liu, Y\. Gao, C\. M\. Meyer, and S\. EgerMoverScore: text generation evaluating with contextualized embeddings and earth mover distance\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 563–578\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, E\. P\. Xing, J\. E\. Gonzalez, I\. Stoica, and H\. ZhangJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2608.14457#S1.p1.1),[§3](https://arxiv.org/html/2608.14457#S3.p5.1)\.
- Zhonget al\.\(2021\)M\. Zhong, D\. Yin, T\. Yu, A\. Zaidi, M\. Mutuma, R\. Jha, A\. H\. Awadallah, A\. Celikyilmaz, Y\. Liu, X\. Qiu, and D\. RadevQMSum: a new benchmark for query\-based multi\-domain meeting summarization\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 5905–5921\.Cited by:[§2](https://arxiv.org/html/2608.14457#S2.p2.1)\.

MetricThr\.RangeOverlap\-basedROUGE\-1 \(P / R / F1\)0\.05\[0,1\]\[0,1\]ROUGE\-2 \(P / R / F1\)0\.03\[0,1\]\[0,1\]ROUGE\-L \(P / R / F1\)0\.05\[0,1\]\[0,1\]BLEU2\.0\[0,100\]\[0,100\]chrF\+\+2\.0\[0,100\]\[0,100\]METEOR0\.05\[0,1\]\[0,1\]CIDEr0\.05\[0,∼10\]\[0,\{\\sim\}10\]DataStats \(surface\)Extractive coverage0\.05\[0,1\]\[0,1\]Extractive density0\.1\[0,∼∞\)\[0,\{\\sim\}\\infty\)Compression ratio0\.1\[0,∼∞\)\[0,\{\\sim\}\\infty\)Summary length \(words\)5\.0\[0,∞\)\[0,\\infty\)% novel 1\-grams0\.05\[0,1\]\[0,1\]% novel 2\-grams0\.05\[0,1\]\[0,1\]% novel 3\-grams0\.05\[0,1\]\[0,1\]% rep\. 1\-grams in summary0\.01\[0,1\]\[0,1\]% rep\. 2\-grams in summary0\.01\[0,1\]\[0,1\]% rep\. 3\-grams in summary0\.01\[0,1\]\[0,1\]Pairwise lengthLength \(word\-count diff\.\)5\.0\[0,∞\)\[0,\\infty\)
\(a\)Lexical metrics\.MetricThr\.RangeBERTScore \(P / R / F1\)0\.02\[0,1\]\[0,1\]SUPERT0\.05\[0,1\]\[0,1\]SummaQA \(avg\. prob\.\)0\.05\[0,1\]\[0,1\]SummaQA \(avg\. F\-score\)0\.05\[0,1\]\[0,1\]BLANC0\.02\[−1,1\]\[\-1,1\]
\(b\)Non\-lexical \(embedding\- and model\-based\) metrics\.
MetricThr\.RangeRaw countsWords10\.0\[0,∞\)\[0,\\infty\)Sentences1\.0\[0,∞\)\[0,\\infty\)Verb phrases2\.0\[0,∞\)\[0,\\infty\)Clauses2\.0\[0,∞\)\[0,\\infty\)T\-units1\.0\[0,∞\)\[0,\\infty\)Dependent clauses1\.0\[0,∞\)\[0,\\infty\)Complex T\-units1\.0\[0,∞\)\[0,\\infty\)Coordinate phrases1\.0\[0,∞\)\[0,\\infty\)Complex nominals2\.0\[0,∞\)\[0,\\infty\)RatiosWords / sentence3\.0\[0,∞\)\[0,\\infty\)Words / T\-unit3\.0\[0,∞\)\[0,\\infty\)Words / clause2\.0\[0,∞\)\[0,\\infty\)Clauses / sentence0\.5\[0,∞\)\[0,\\infty\)Verb phrases / T\-unit0\.5\[0,∞\)\[0,\\infty\)Clauses / T\-unit0\.5\[0,∞\)\[0,\\infty\)Dep\. clauses / clause0\.1\[0,∞\)\[0,\\infty\)Dep\. clauses / T\-unit0\.2\[0,∞\)\[0,\\infty\)T\-units / sentence0\.3\[0,∞\)\[0,\\infty\)Complex T\-units / T\-unit0\.2\[0,∞\)\[0,\\infty\)Coord\. phrases / T\-unit0\.2\[0,∞\)\[0,\\infty\)Coord\. phrases / clause0\.1\[0,∞\)\[0,\\infty\)Complex nominals / T\-unit0\.5\[0,∞\)\[0,\\infty\)Complex nominals / clause0\.3\[0,∞\)\[0,\\infty\)
\(c\)Syntactic complexity metrics\.MetricThr\.RangeLLM Judge \(per\-dimension\)Relevance0\.5\[1,5\]\[1,5\]Coherence0\.5\[1,5\]\[1,5\]Consistency0\.5\[1,5\]\[1,5\]Fluency0\.5\[1,5\]\[1,5\]Informativeness0\.5\[1,5\]\[1,5\]Overall0\.5\[1,5\]\[1,5\]OtherFActScore0\.1\[0,1\]\[0,1\]
\(d\)LLM\-based metrics\.

Table 5\.Per\-metric similarity thresholds used in the agreement analysis \([Sections6](https://arxiv.org/html/2608.14457#S6)and[3](https://arxiv.org/html/2608.14457#S5.T3)\)\. When\|metric​\(A\)−metric​\(B\)\|\|\\text\{metric\}\(A\)\-\\text\{metric\}\(B\)\|is below the listed threshold, the metric’s decision is counted as “Neither\.” Thresholds reflect the typical scale and variance of each metric\.Prompt: LLM Judge

\#\#\#TaskDescription:

Aninstruction\(mightincludeanInputinsideit\),aresponsetoevaluate,andascorerubricrepresentinganevaluationcriteriaaregiven\.

1\.Writeadetailedfeedbackthatassessesthequalityoftheresponsestrictlybasedonthegivenscorerubric,notevaluatingingeneral\.

2\.Afterwritingafeedback,writeascorethatisanintegerbetween1and5\.Youshouldrefertothescorerubric\.

3\.Theoutputformatshouldlookasfollows:"Feedback:\(writeafeedbackforcriteria\)\[RESULT\]\(anintegernumberbetween1and5\)"

4\.Pleasedonotgenerateanyotheropening,closing,orexplanations\.

\#\#\#Instruction:

Youareevaluatingasummaryfromtheperspectiveofaspecificperson\.Considertheirbackgroundandinformationneedswhenassessingquality\.

Annotatorprofile:

\-Role:\{role\}

\-Domain:\{domain\}

\-Informationneeds:\{info\_needs\}

\-Query:\{query\}

Evaluatethequalityofthefollowingsummaryonthedimensionof\{dimension\},fromthisperson’sperspective\.

Summary:

\{summary\}

\#\#\#Responsetoevaluate:

\{summary\}

\#\#\#ScoreRubric:

\{rubric\}

Table 6\.Prompt template for LLM Judges\. Score rubrics in[Table7](https://arxiv.org/html/2608.14457#A0.T7)\. Source is also provided in prompt for consistency and relevance\.Score Rubrics

Coherence
Isthesummarywell\-organizedandeasytoread?

\[1\]Thesummaryisincoherentandverydifficulttofollow\.

\[2\]Thesummaryhaspoororganizationwithfrequentdisjointedtransitions\.

\[3\]Thesummaryissomewhatorganizedbuthasnoticeablestructuralissues\.

\[4\]Thesummaryiswell\-organizedwithonlyminorflowissues\.

\[5\]Thesummaryisexcellentlystructuredandflowsnaturallythroughout\.

Consistency
Isthesummaryfactuallyconsistentwiththesourcedocument?

\[1\]Thesummarycontainsmajorfactualerrorsorhallucinationsnotfoundinthesource\.

\[2\]Thesummarycontainsseveralfactualinconsistencieswiththesource\.

\[3\]Thesummaryismostlyconsistentbutcontainsafewminorfactualerrors\.

\[4\]Thesummaryislargelyconsistentwithonlynegligibleinaccuracies\.

\[5\]Allinformationinthesummaryisfullyconsistentwiththesourcedocument\.

Fluency
Isthesummarygrammaticallycorrectandwell\-written?

\[1\]Thesummaryhasmanygrammaticalerrorsandispoorlywritten\.

\[2\]Thesummaryhasfrequentgrammaticalissuesthathinderreadability\.

\[3\]Thesummaryhasoccasionalgrammaticalerrorsbutisgenerallyreadable\.

\[4\]Thesummaryiswell\-writtenwithonlyminorgrammaticalissues\.

\[5\]Thesummaryhasflawlessgrammarandexcellentwritingquality\.

Relevance
Doesthesummarycapturethekeyinformationfromthesource?

\[1\]Thesummaryiscompletelyirrelevantandfailstocaptureanykeyinformationfromthesourcedocument\.

\[2\]Thesummarycapturesverylittlekeyinformationandmissesmostimportantpoints\.

\[3\]Thesummarycapturessomekeyinformationbutmissesseveralimportantpoints\.

\[4\]Thesummarycapturesmostkeyinformationwithonlyminoromissions\.

\[5\]Thesummarycapturesallkeypointsandessentialinformationfromthesource\.

Informativeness
Howusefulandinformativeisthesummarytothisspecificperson,giventheirbackgroundandinformationneeds?

\[1\]Thesummaryprovidesnousefulinformationforthisperson’sspecificneedsandbackground\.Itfailstoaddresstheirdomainorinformationrequirements\.

\[2\]Thesummaryprovidesverylittleinformationrelevanttothisperson’sneeds\.Itlargelymissescontentthatwouldbeusefulgiventheirroleanddomainexpertise\.

\[3\]Thesummaryprovidessomeinformationrelevanttothisperson’sneedsbutlacksimportantdomain\-specificdetailsorfailstoaddresstheirstatedinformationrequirements\.

\[4\]Thesummaryisinformativeforthisperson’sneeds,coveringmostrelevantdetailsfortheirroleanddomain,withonlyminorgapsinaddressingtheirinformationrequirements\.

\[5\]Thesummaryishighlyinformativeforthisperson’sspecificneeds,thoroughlyaddressingtheirdomainexpertiseandinformationrequirementswithrelevantdetailandappropriatecontext\.

Table 7\.Score rubrics for prompt in[Table6](https://arxiv.org/html/2608.14457#A0.T6)\.Prompt: LLM Judge Annotator

\#\#\#TaskDescription:

Youareevaluatingasummaryfromtheperspectiveofaspecificannotator\.Considertheirbackgroundandinformationneedswhenassessingquality\.

1\.Writeadetailedfeedbackthatassessesstrictlywhetherthesummarybestaddressestheannotator’squery,notthefidelityorthestyle\.

2\.Afterwritingafeedback,writeascorethatisanintegerbetween1and5\.Youshouldrefertothescorerubric\.

3\.Theoutputformatshouldlookasfollows:"Feedback:\(writeafeedbackforcriteria\)\[RESULT\]\(anintegernumberbetween1and5\)"

4\.Pleasedonotgenerateanyotheropening,closing,orexplanations\.

\#\#\#Introduction

Automaticsummarizationtoolscondenselargeamountsoftextintoshorterversions,highlightingthemostimportantpoints\.

But"important"issubjective\-whatmatterstoonereadermightnotmattertoanother\.

Thisstudyexploreswhatindividualreadersactuallywantfromsummariesofscientificpapers,andhowwecanmeasurewhetherasummarywasusefultothem\.

\*\*We’refocusedonwhetherthesummaryanswersyourquery,notthestyleorfidelityofthesummary\.\*\*

\#\#\#Annotatorprofile:

\-Role:\{role\}

\-Domain:\{domain\}

\-Informationneeds:\{info\_needs\}

\#\#\#Query:

\{query\}

\#\#\#Responsetoevaluate:

\{summary\}

\#\#\#ScoreRubric:

Howwelldoesthesummaryaddresstheannotator’sspecificquery,giventheirbackgroundandinformationneeds?

\[1\]Thesummarydoesnotaddresstheannotator’squeryatall\.Itprovidesnoinformationthatwouldhelpanswerwhattheyaskedabout\.

\[2\]Thesummarybarelyaddressesthequery,touchingonitonlytangentiallyorprovidingverylittleoftherequestedinformation\.

\[3\]Thesummarypartiallyaddressesthequery,providingsomerelevantinformationbutmissingimportantaspectsofwhatwasasked\.

\[4\]Thesummaryaddressesthequerywell,coveringmostofwhatwasaskedwithonlyminorgaps\.

\[5\]Thesummaryfullyanddirectlyaddressestheannotator’squery,providingtherequestedinformationclearlyandcompletely\.

Table 8\.Prompt template for LLM Judge Annotator, in which the prompt reflects the instructions given to the human annotators\. Score rubrics in[Table7](https://arxiv.org/html/2608.14457#A0.T7)\. Source is also provided in prompt for consistency and relevance\.Prompt: Persona Precision

Step: Extract
Breakdownthefollowingsummaryintoalistofindependentinformationalnuggets\.Eachnuggetshouldbeasingle,self\-containedpieceofinformation\-\-aminimal,atomicstatementthatconveysonefactorclaim\.

\#\#Summary

\{summary\}

\#\#Instructions

Listeachinformationalnuggetonitsownline,prefixedwith"\-"\.OutputONLYthelistofnuggets,nothingelse\.

Exampleformat:

\-Thestudyusedarandomizedcontrolledtrialdesign\.

\-Thesampleincluded500participantsfromthreehospitals\.

\-Treatmentgroupshowed23%improvementovercontrol\.

Step: Relevance
Determinewhetherthefollowinginformationalnuggetfromasummaryisrelevanttothegivenpersonaandtheirquery\.

Anuggetis"relevant"ifitaddressesthepersona’sinformationneeds,relatestotheirdomainofexpertise,orhelpsanswertheirquery\.Anuggetis"notrelevant"ifitisunrelatedtowhatthispersonawouldcareabout\.

\#\#Persona

\-Role:\{role\}

\-Domain:\{domain\}

\-Informationneeds:\{info\_needs\}

\#\#Query

\{query\}

\#\#Nugget

\{nugget\}

\#\#Instructions

Isthisnuggetrelevanttothepersona’sneedsandquery?RespondwithONLY"relevant"or"notrelevant"\.

Table 9\.Prompts for Persona Precision metric\.Prompt: Persona Recall

Step: Extract
Breakdownthefollowingsummaryintoalistofindependentinformationalnuggets\.Eachnuggetshouldbeasingle,self\-containedpieceofinformation\-\-aminimal,atomicstatementthatconveysonefactorclaim\.

\#\#Summary

\{summary\}

\#\#Instructions

Listeachinformationalnuggetonitsownline,prefixedwith"\-"\.OutputONLYthelistofnuggets,nothingelse\.

Exampleformat:

\-Thestudyusedarandomizedcontrolledtrialdesign\.

\-Thesampleincluded500participantsfromthreehospitals\.

\-Treatmentgroupshowed23%improvementovercontrol\.

Step: Requirements
Youaregivenapersonadescriptionandtheirqueryaboutasourcedocument\.Generatealistofinformationalnuggetrequirements\-\-specificpiecesofinformationthatagoodsummarySHOULDcontaintoservethispersona’sneeds\.

Eachrequirementshouldbeanatomic,specificpieceofinformationthatthispersonawouldexpectorneedfromasummaryofthesourcedocument\.

\#\#Persona

\-Role:\{role\}

\-Domain:\{domain\}

\-Informationneeds:\{info\_needs\}

\#\#Query

\{query\}

\#\#SourceDocument

\{source\}

\#\#Instructions

Basedonthepersona’sbackground,informationneeds,andquery,listthekeyinformationalnuggetsthatagoodsummaryshouldinclude\.Eachnuggetshouldbeasingle,specificpieceofinformationfromthesourcedocument\.

Listeachrequirementonitsownline,prefixedwith"\-"\.OutputONLYthelistofrequirements,nothingelse\.

Exampleformat:

\-Themainmethodologyusedinthestudy\.

\-Keyquantitativeresultsandeffectsizes\.

\-Limitationsthataffectpracticalapplication\.

Step: Coverage
Determinewhetherthefollowinginformationalrequirementiscoveredbyanyofthesummarynuggetsbelow\.

Arequirementis"covered"ifanyofthesummarynuggetsaddressthesameinformation,evenifphraseddifferently\.

\#\#Requirement

\{requirement\}

\#\#SummaryNuggets

\{nuggets\}

\#\#Instructions

Istherequirementaddressedbyanyofthesummarynuggetsabove?RespondwithONLY"covered"or"notcovered"\.

Table 10\.Prompts for Persona Recall metric\.Prompt: Perturbation Prompts

Test: Lengthen
Youaregivenasummaryandthesourcedocumentitwasbasedon\.

Rewritethesummarytobelongerandmoredetailed\.YouMUSTfollowtheserules:

\-DoNOTaddanynewinformationthatisnotalreadypresentinordirectlyinferablefromtheoriginalsummary\.

\-Keepalloriginalfacts,claims,andfindingsintact\.

\-Expandbyelaboratingonexistingpoints,addingtransitionalphrases,andusingmoredescriptivelanguage\.

\-Therewrittensummaryshouldbeapproximately50%longerthantheoriginal\.

Sourcedocument:

\{source\}

Originalsummary:

\{summary\}

Rewritten\(longer\)summary:

Test: Shorten
Youaregivenasummaryandthesourcedocumentitwasbasedon\.

Rewritethesummarytobeshorterandmoreconcise\.YouMUSTfollowtheserules:

\-DoNOTremoveanykeyinformation,facts,claims,orfindingsfromtheoriginalsummary\.

\-Preserveallsubstantivecontentwhileusingfewerwords\.

\-Removeredundancy,simplifyphrasing,andtightensentencestructure\.

\-Therewrittensummaryshouldbeapproximately50%shorterthantheoriginal\.

Sourcedocument:

\{source\}

Originalsummary:

\{summary\}

Rewritten\(shorter\)summary:

Test: Different Audience
Youaregivenasummaryofasourcedocument\.Thesummarywasoriginallywrittenfor:\{original\_audience\}\.

Rewritethissummaryforadifferenttargetaudience:\{target\_audience\}\.

Youshould:

\-Adjustthelanguage,terminology,andlevelofdetailtobeappropriateforthetargetaudience\.

\-Youmaychangewhichinformationisemphasizedorhowconceptsareexplained\.

\-Baseyourrewriteonthesourcedocument,notjusttheoriginalsummary\.

\-Therewriteshouldberoughlythesamelengthastheoriginalsummary\.

Sourcedocument:

\{source\}

Originalsummary\(writtenfor\{original\_audience\}\):

\{summary\}

Rewrittensummary\(for\{target\_audience\}\):

Table 11\.Prompts for the synthetic perturbations tests, described in[Section3](https://arxiv.org/html/2608.14457#S3)\.