Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

# Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Source: [https://arxiv.org/html/2607.10806](https://arxiv.org/html/2607.10806)
Praveenkumar Katwe1, Rakesh Chandra Balabantaray1, Kali Prasad Vittala2 1Department of Computer Science and Engineering, International Institute of Information Technology, Bhubaneswar, India 2Salesforce India Pvt Ltd, Bengaluru, India c121007@iiit\-bh\.ac\.in

###### Abstract

Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface\-level metrics like ROUGE\. We introduce Reference Abstraction \(RA\), Summary Abstraction \(SA\), and Abstraction Ratio \(AR\)—a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text\. The formulation uses the harmonic mean of document lengths modulated by a cubic non\-overlap factor, yielding dimensionally consistent, bounded output with non\-linear sensitivity to the extractive\-abstractive boundary\. Evaluation on 100 XSUM documents across four summarization models \(BART\-large\-cnn, Pegasus\-xsum, DistilBart, MT5\-small\) demonstrates that the metrics successfully discriminate between extractive models \(SA≈0\.12\\approx 0\.12–0\.260\.26\) and abstractive models \(SA≈0\.96\\approx 0\.96–1\.771\.77\), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination\. Code and results are available at[https://github\.com/katweNLP/AbstractionStudy](https://github.com/katweNLP/AbstractionStudy)\.

## 1Introduction

Abstractive summarization generates novel text that captures the gist of a source document\. Unlike extractive methods that copy sentences verbatim, abstractive models paraphrase, generalize, and sometimes hallucinate—introducing content not grounded in the source\. While fluency and informativeness have well\-established metrics \(ROUGE\[[1](https://arxiv.org/html/2607.10806#bib.bib1)\], BERTScore\[[2](https://arxiv.org/html/2607.10806#bib.bib2)\]\),*abstractiveness itself*—the degree to which a summary diverges from extractive copying—has lacked a principled quantitative measure\.

Prior work has used novel n\-gram ratio\[[3](https://arxiv.org/html/2607.10806#bib.bib3)\]as a proxy, but this binary token\-level measure does not account for document length asymmetry or provide graduated discrimination at different overlap levels\. We propose three complementary metrics—RA, SA, and AR—that address these limitations through a formulation combining harmonic\-mean length weighting with cubic non\-overlap amplification\.

These metrics serve as a*screening tool*: high RA indicates that the reference itself is highly abstractive \(and potentially hallucinated\); mismatched AR values flag summaries requiring manual evaluation\. The metrics motivate and precede the Entity Hallucination Index \(EHI\)\[[4](https://arxiv.org/html/2607.10806#bib.bib4)\], which provides fine\-grained hallucination decomposition\.

## 2Related Work

### 2\.1Reference\-Based Quality Metrics

The dominant paradigm for summarization evaluation relies on comparing generated text against human\-written references\. ROUGE\[[1](https://arxiv.org/html/2607.10806#bib.bib1)\]measures n\-gram overlap between generated and reference summaries, providing recall\-oriented quality scores\. While ROUGE remains the de facto standard for benchmarking, it fundamentally rewards extractive copying—a model that reproduces the reference verbatim achieves a perfect score regardless of whether it faithfully represents the source document\. BERTScore\[[2](https://arxiv.org/html/2607.10806#bib.bib2)\]addresses surface\-level matching limitations by computing semantic similarity via contextual embeddings, but remains reference\-based: it measures how well the generated summary matches human expectations, not whether it is grounded in the source\.

A critical limitation shared by all reference\-based metrics is their inability to detect*faithfulness*violations\. A summary can achieve high ROUGE by capturing the same content choices as the reference while simultaneously introducing hallucinated details not present in the source\. This disconnect between quality and faithfulness was demonstrated empirically by Fabbri et al\.\[[5](https://arxiv.org/html/2607.10806#bib.bib5)\], who showed poor correlation between automatic metrics \(including ROUGE and BERTScore\) and human judgments of faithfulness in the SummEval benchmark\.

### 2\.2Faithfulness and Factuality

Maynez et al\.\[[6](https://arxiv.org/html/2607.10806#bib.bib6)\]provided the first large\-scale study of hallucination in abstractive summarization, finding that over 30% of generated summaries contain hallucinated content \(intrinsic or extrinsic\)\. Crucially, they demonstrated that ROUGE scores do not correlate with faithfulness—models can score well on ROUGE while producing unfaithful summaries\. This finding motivates source\-grounding metrics that assess the relationship between generated text and the*source document*rather than a reference\.

FactCC\[[7](https://arxiv.org/html/2607.10806#bib.bib7)\]addresses faithfulness through natural language inference \(NLI\), classifying each generated sentence as entailed or not entailed by the source\. While effective as a binary detector, FactCC provides no gradation—it cannot distinguish a mildly paraphrased sentence from a completely fabricated one\. Nan et al\.\[[8](https://arxiv.org/html/2607.10806#bib.bib8)\]proposed entity\-level factual consistency, measuring whether named entities in the summary are grounded in the source\. This provides interpretable signals but reduces faithfulness to a single entity\-overlap score without decomposition by entity type or severity\.

### 2\.3Abstractiveness Proxies

Narayan et al\.\[[3](https://arxiv.org/html/2607.10806#bib.bib3)\]introduced the percentage of novel n\-grams as a simple proxy for abstractiveness in their XSUM dataset paper\. While useful for dataset characterization, this measure is binary at the token level \(each n\-gram is either novel or not\) and does not account for the length asymmetry between source documents and summaries\. A 20\-word summary with 50% novel unigrams is treated identically whether the source is 100 words or 10,000 words\.

### 2\.4Positioning of Our Work

Our proposed metrics \(RA, SA, AR\) occupy a distinct niche in this landscape\. Unlike ROUGE and BERTScore, they are*source\-grounding*metrics that measure the relationship between generated text and the source document\. Unlike FactCC, they provide*continuous scores*with interpretable thresholds rather than binary classifications\. Unlike novel n\-gram ratio, they incorporate*length normalization*via the harmonic mean and*non\-linear sensitivity*via cubic amplification\. Table[1](https://arxiv.org/html/2607.10806#S2.T1)summarizes the positioning\.

Table 1:Comparison of summarization evaluation approaches

## 3Metric Definitions

### 3\.1Variable Definitions

Table 2:Variable definitions with domains
### 3\.2Formulas

RA=HM​\(R,O\)×\(1−cR\)3=2​R​OR\+O⋅\(1−cR\)3\\text\{RA\}=\\text\{HM\}\(R,O\)\\times\(1\-c\_\{R\}\)^\{3\}=\\frac\{2RO\}\{R\+O\}\\cdot\(1\-c\_\{R\}\)^\{3\}\(1\)
SA=HM​\(S,O\)×\(1−cS\)3=2​S​OS\+O⋅\(1−cS\)3\\text\{SA\}=\\text\{HM\}\(S,O\)\\times\(1\-c\_\{S\}\)^\{3\}=\\frac\{2SO\}\{S\+O\}\\cdot\(1\-c\_\{S\}\)^\{3\}\(2\)
AR=\(1−cS\)3\(1−cR\)3\+ε\\text\{AR\}=\\frac\{\(1\-c\_\{S\}\)^\{3\}\}\{\(1\-c\_\{R\}\)^\{3\}\+\\varepsilon\}\(3\)
Note on AR formulation:While RA and SA incorporate the harmonic mean to provide absolute abstractiveness scores \(in word\-count units\), AR is defined as a*purely dimensionless ratio*using only the normalized non\-overlap factors\. This design choice ensures:

1. 1\.AR is interpretable around the threshold of 1\.0 \(AR≈1\\approx 1means equal abstractiveness\),
2. 2\.AR is independent of absolute document length \(HM\(R,O\)\(R,O\)and HM\(S,O\)\(S,O\)would not cancel cleanly sinceR≠SR\\neq S\), and
3. 3\.AR remains bounded and comparable across document pairs of different sizes\.

### 3\.3Design Rationale

Table 3:Term\-by\-term justification
### 3\.4Mathematical Properties

- •Dimensional consistency:HM\(R,O\)\(R,O\)has units of word count;\(1−cR\)3\(1\-c\_\{R\}\)^\{3\}is dimensionless\. RA and SA have units of word count—interpretable and consistent\. AR is purely dimensionless\.
- •Monotonic behavior:High overlap⇒cR→1⇒\(1−cR\)3→0⇒\\Rightarrow c\_\{R\}\\to 1\\Rightarrow\(1\-c\_\{R\}\)^\{3\}\\to 0\\RightarrowRA→0\\to 0\(extractive\)\. Low overlap⇒cR→0⇒\\Rightarrow c\_\{R\}\\to 0\\RightarrowRA==HM\(R,O\)\(R,O\)\(maximum abstractiveness\)\.
- •Bounded:RA∈\[0,HM​\(R,O\)\]\\in\[0,\\text\{HM\}\(R,O\)\]\. SA∈\[0,HM​\(S,O\)\]\\in\[0,\\text\{HM\}\(S,O\)\]\. AR∈\[0,∞\)\\in\[0,\\infty\)with AR=1=1at equal abstractiveness\.
- •No division by zero:\(1−cR\)3\(1\-c\_\{R\}\)^\{3\}is a multiplier in RA/SA, never a denominator\. Theε\\varepsilonin AR guards against the edge case where the reference is fully extractive \(cR=1c\_\{R\}=1\)\.
- •Interpretability of AR:Because AR uses only the dimensionless\(1−c\)3\(1\-c\)^\{3\}factors, it directly compares the non\-overlap*proportions*raised to the cubic power\. AR=1=1means identical abstractiveness regardless of document length\. This separates the*comparison*function \(AR\) from the*measurement*function \(RA, SA\)\.

## 4Experimental Setup

### 4\.1Datasets

We evaluate on two datasets with contrasting reference styles:

- •XSUM\[[3](https://arxiv.org/html/2607.10806#bib.bib3)\]: 50 validation documents with highly abstractive single\-sentence references \(meancR=0\.61c\_\{R\}=0\.61, i\.e\., 39% novel content\)\. Chosen to test AR<1<1behavior\.
- •CNN/DailyMail: 50 validation documents with extractive multi\-sentence references \(meancR=0\.83c\_\{R\}=0\.83, i\.e\., only 17% novel content\)\. Chosen to test AR\>1\>1behavior and hallucination detection\.

### 4\.2Models

Four summarization models spanning the extractive\-abstractive spectrum:

1. 1\.facebook/bart\-large\-cnn\[[9](https://arxiv.org/html/2607.10806#bib.bib9)\]: Pre\-trained on CNN/DailyMail \(extractive\-biased\)
2. 2\.google/pegasus\-xsum\[[10](https://arxiv.org/html/2607.10806#bib.bib10)\]: Pre\-trained specifically for extreme summarization
3. 3\.sshleifer/distilbart\-cnn\-12\-6: Distilled BART \(extractive\-biased\)
4. 4\.google/mt5\-small: Multilingual T5 \(zero\-shot, tends toward short abstractive output\)

### 4\.3Overlap Computation

For each document\-summary pair, we compute normalized overlap at three granularities:

- •Unigram \(c1c\_\{1\}\):Token\-level multiset intersection normalized by summary length
- •Bigram \(c2c\_\{2\}\):Bigram intersection normalized by number of bigrams in summary
- •LCS \(cLc\_\{L\}\):Longest common subsequence length normalized by summary length

## 5Results

### 5\.1Reference Abstraction \(RA\): Dataset Characterization

RA characterizes the inherent abstractiveness of gold\-standard references:

Table 4:Reference Abstraction by dataset \(50 articles each\)XSUM references are substantially more abstractive \(RA1= 3\.55\) than CNN/DM references \(RA1= 0\.61\), confirming the known design difference between these benchmarks\.

![Refer to caption](https://arxiv.org/html/2607.10806v1/results/fig_RA_dual_dataset.png)Figure 1:Reference Abstraction \(RA\) per article: XSUM \(left, abstractive\) vs CNN/DailyMail \(right, extractive\)\.
### 5\.2Summary Abstraction \(SA\)

Table 5:Summary Abstraction by model and dataset \(50 articles each\)SA clearly separates abstractive models \(Pegasus, MT5\) from extractive ones \(BART, DistilBart\) on both datasets\. Notably, BART/DistilBart maintain high extraction \(cS≈0\.97c\_\{S\}\\approx 0\.97\) regardless of whether the dataset expects abstraction \(XSUM\) or extraction \(CNN/DM\)\.

![Refer to caption](https://arxiv.org/html/2607.10806v1/results/fig_SA_dual_dataset.png)Figure 2:Summary Abstraction across articles: XSUM \(top row\) vs CNN/DailyMail \(bottom row\)\.
### 5\.3Abstraction Ratio \(AR\): The Key Finding

AR=\(1−cS\)3/\(1−cR\)3=\(1\-c\_\{S\}\)^\{3\}/\(1\-c\_\{R\}\)^\{3\}measures how the generated summary’s abstractiveness compares to the reference\. The dual\-dataset evaluation reveals the full AR spectrum:

Table 6:Abstraction Ratio \(median\): AR<1<1= extractive, AR≈1\\approx 1= matched, AR\>1\>1= over\-abstractiveKey observations:

- •OnXSUM\(abstractive references\): All models have AR<1<1—none achieve the reference’s abstraction level\. Pegasus comes closest \(AR2= 0\.92\)\.
- •OnCNN/DailyMail\(extractive references\): Pegasus and MT5 achieveAR\>1\>1, indicating they abstract*more*than the reference\. This flags potential hallucination—the model introduces novel content beyond what the human deemed necessary\.
- •BART/DistilBart remain extractive \(AR≈0\\approx 0\) on both datasets, consistent with their CNN/DM pre\-training bias\.

![Refer to caption](https://arxiv.org/html/2607.10806v1/results/fig_AR_dual_dataset.png)Figure 3:Abstraction Ratio: XSUM \(top, all AR<1<1\) vs CNN/DailyMail \(bottom, abstractive models exceed AR=1=1\)\. Red dashed line marks equal abstractiveness\.
### 5\.4Hallucination Detection via AR

We classify each generated summary based on AR2\(bigram\) thresholds:

- •AR\>2\.0\>2\.0:High hallucination risk\(model far exceeds reference abstraction\)
- •1\.2<1\.2<AR≤2\.0\\leq 2\.0: Moderate risk
- •0\.8≤0\.8\\leqAR≤1\.2\\leq 1\.2: Balanced \(ideal\)
- •AR<0\.8<0\.8: Extractive \(insufficient abstraction\)

Table 7:Hallucination risk detection via AR2threshold \(out of 50 articles per cell\)On CNN/DailyMail,Pegasus triggers high hallucination risk on 44% of articles\(22/50\)—it generates substantially more novel content than the extractive references warrant\. This correctly identifies summaries that require human verification: the model is abstracting beyond what the gold standard supports, potentially introducing unfaithful content\.

![Refer to caption](https://arxiv.org/html/2607.10806v1/results/fig_hallucination_detection.png)Figure 4:Hallucination risk classification by model and dataset\. CNN/DM shows significantly more high\-risk flagging for abstractive models\.

## 6Discussion

### 6\.1Key Observations

1. 1\.Models trained on extractive data remain extractive on abstractive benchmarks\.BART\-large\-cnn and DistilBart, both pre\-trained on CNN/DailyMail \(an extractive dataset\), produce summaries with 98% overlap on XSUM\. The SA metric correctly identifies this extractive behavior \(SA≈10\.12\{\}\_\{1\}\\approx 0\.12–0\.260\.26\)\.
2. 2\.Pegasus\-xsum achieves genuinely abstractive output\.With 31% novel content \(cS=0\.69c\_\{S\}=0\.69\), Pegasus matches the abstraction level of human references \(cR=0\.63c\_\{R\}=0\.63\)\. This validates Pegasus’s pre\-training objective of gap\-sentence generation for abstractive tasks\.
3. 3\.High AR indicates the need for manual evaluation\.When AR≫1\\gg 1, the model is far less abstractive than the reference\. This identifies articles where the model failed to abstract—potentially due to domain mismatch, input length, or model limitations\.
4. 4\.Cubic amplification provides sharp discrimination\.The\(1−c\)3\(1\-c\)^\{3\}term makes the metric highly sensitive nearc≈1c\\approx 1\(extractive boundary\)\. The difference betweencS=0\.95c\_\{S\}=0\.95andcS=0\.98c\_\{S\}=0\.98is amplified:\(1−0\.95\)3=0\.000125\(1\-0\.95\)^\{3\}=0\.000125vs\(1−0\.98\)3=0\.000008\(1\-0\.98\)^\{3\}=0\.000008—a 16×\\timesratio from a 3% overlap difference\. This sensitivity is desirable for detecting subtle extractive biases\.

### 6\.2Relationship to Hallucination Detection

A high RA score indicates the reference summary itself contains substantial novel content not grounded in the source—i\.e\., the reference exhibits positive hallucination\. This insight motivates the Entity Hallucination Index \(EHI\)\[[4](https://arxiv.org/html/2607.10806#bib.bib4)\], which decomposes this novelty into beneficial \(positive hallucination\) and harmful \(negative hallucination\) components\. The RA/SA/AR metrics serve as a*first\-order screening tool*: they identify that hallucination exists, while EHI determines whether it is constructive or destructive\.

### 6\.3Complementarity with ROUGE and FactCC

A natural question arises: if ROUGE already evaluates summary quality, why is AR needed? The answer lies in a fundamental distinction between*quality*\(match to reference\) and*faithfulness*\(grounding in source\)\.

ROUGE measures quality relative to the reference\.High ROUGE indicates that the generated summary captures similar content to the human\-written reference\. However, high ROUGE does not imply faithfulness—both the reference and the generated summary may contain content not present in the source \(hallucination\), and ROUGE rewards this agreement\.

AR measures abstractiveness relative to the source\.High AR indicates that the generated summary introduces substantially more novel content \(relative to the source\) than the reference does\. This flags over\-abstraction risk—the model is going beyond what the human annotator deemed necessary\.

These metrics are complementary, not redundant\.A model can simultaneously achieve:

- •*High ROUGE and low AR*: The model matches the reference well and abstracts at a similar level—ideal behavior\.
- •*High ROUGE and high AR*: The model captures the reference’s content choices but introduces even more novel content—hallucination hiding behind good quality scores\. This is precisely the dangerous case that ROUGE alone cannot detect\.
- •*Low ROUGE and high AR*: The model diverges from both reference and source—clear quality failure\.
- •*Low ROUGE and low AR*: The model is overly extractive relative to an abstractive reference—captures source content but misses the reference’s abstraction\.

Compared to FactCC\[[7](https://arxiv.org/html/2607.10806#bib.bib7)\], which provides a binary entailment judgment \(faithful/unfaithful\), AR offers a continuous score with an interpretable threshold at AR=1=1\. FactCC cannot distinguish between a summary that is 5% over\-abstractive and one that is 50% over\-abstractive\. AR provides this gradation, enabling prioritized manual review: summaries with AR\>2\>2warrant immediate attention, while those with1\.2<1\.2<AR<2<2require lighter review\.

Our empirical results confirm this complementarity directly:

Table 8:ROUGE vs AR: Empirical demonstration that quality≠\\neqfaithfulnessKey observation:On CNN/DailyMail, BART achieves the highest ROUGE \(0\.393\) with zero hallucination risk \(AR≈0\\approx 0\)—it copies from the source faithfully\. Meanwhile, Pegasus achieves lower ROUGE \(0\.197\) but triggers HIGH hallucination risk \(AR = 3\.74\)—it introduces novel content far beyond what the reference warrants\. ROUGE alone would rank BART as superior; AR reveals that Pegasus’s lower ROUGE is partly because it is*hallucinating*beyond the reference, not because it is a weaker model\.

The Pearson correlation between ROUGE\-1 and AR1across all CNN/DM samples isr=−0\.28r=\-0\.28\(p<0\.001p<0\.001\), confirming that these metrics capture independent dimensions\. On XSUM the correlation isr=0\.10r=0\.10\(not significant\)—further evidence of orthogonality\.

![Refer to caption](https://arxiv.org/html/2607.10806v1/results/fig_rouge_vs_ar.png)Figure 5:ROUGE\-1 vs AR scatter: each point is one article\. ROUGE \(x\-axis\) measures quality; AR \(y\-axis\) measures abstractiveness relative to reference\. The two dimensions are largely independent\.Without AR, Pegasus’s over\-abstractive summaries on CNN/DM would pass quality checks based on ROUGE alone\. AR provides the complementary faithfulness signal that ROUGE cannot\.

### 6\.4Why AR Uses Only\(1−c\)3\(1\-c\)^\{3\}Without HM

A natural question arises: why not define AR==RA/SA directly? The reason is that RA and SA include the harmonic mean factor HM\(R,O\)\(R,O\)and HM\(S,O\)\(S,O\)respectively, which are in units of word count\. SinceR≠SR\\neq Sin general, these do not cancel cleanly in the ratio:

RASA=HM​\(R,O\)HM​\(S,O\)×\(1−cR\)3\(1−cS\)3\\frac\{\\text\{RA\}\}\{\\text\{SA\}\}=\\frac\{\\text\{HM\}\(R,O\)\}\{\\text\{HM\}\(S,O\)\}\\times\\frac\{\(1\-c\_\{R\}\)^\{3\}\}\{\(1\-c\_\{S\}\)^\{3\}\}
The first factorHM​\(R,O\)/HM​\(S,O\)\\text\{HM\}\(R,O\)/\\text\{HM\}\(S,O\)is a residual length ratio that confounds the abstractiveness comparison\. By defining AR using only the\(1−c\)3\(1\-c\)^\{3\}terms, we isolate the pure abstractiveness comparison, making AR:

- •Fully dimensionless \(ratio of two \[0,1\] values raised to the same power\),
- •Independent of absolute document or summary length, and
- •Interpretable with a universal threshold at AR=1=1\.

RA and SA retain the HM factor for use cases where absolute abstractiveness magnitude matters \(e\.g\., comparing across datasets of different scales\)\. AR strips it for direct model\-vs\-reference comparison\.

### 6\.5Limitations

- •The metric measures abstractiveness at the surface level \(word/n\-gram overlap\)\. Semantic paraphrasing that preserves meaning but changes surface form is counted as abstraction, which is the intended behavior for measuring divergence from extractive copying\.
- •The choice of cubic exponent \(3\) is empirically motivated rather than axiomatically derived\. It represents a design choice validated through ablation rather than theoretical necessity\.
- •For highly extractive models \(cS\>0\.95c\_\{S\}\>0\.95\), AR approaches zero rapidly due to the cubic—small differences in overlap produce large AR differences\. This is by design \(sensitivity at the extractive boundary\) but means AR is most informative for models in the abstractive range \(cS<0\.8c\_\{S\}<0\.8\)\.

## 7Conclusion

We present Reference Abstraction \(RA\), Summary Abstraction \(SA\), and Abstraction Ratio \(AR\) as principled heuristic metrics for quantifying abstractiveness in text summarization\. The formulation—harmonic mean of document lengths modulated by normalized cubic non\-overlap—achieves dimensional consistency, bounded output, and non\-linear sensitivity to the extractive\-abstractive boundary\. Evaluation across four models on XSUM demonstrates clear discrimination between extractive \(BART, DistilBart\) and abstractive \(Pegasus, MT5\) systems\. The metrics provide a screening tool for identifying potentially hallucinated summaries, motivating deeper analysis via entity and relation hallucination metrics\.

### Code Availability

## References

- \[1\]Chin\-Yew Lin\.ROUGE: A package for automatic evaluation of summaries\.InText Summarization Branches Out, pages 74–81\. Association for Computational Linguistics, 2004\.
- \[2\]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q\. Weinberger, and Yoav Artzi\.BERTScore: Evaluating text generation with BERT\.arXiv preprint arXiv:1904\.09675, 2019\.
- \[3\]Shashi Narayan, Shay B\. Cohen, and Mirella Lapata\.Don’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, 2018\.
- \[4\]Praveenkumar Katwe, Rakesh Chandra Balabantaray, and Kali Prasad Vittala\.Entity hallucination index in abstractive summarization – a metric\.In2023 International Conference on Communication, Circuits, and Systems \(IC3S\), pages 1–5\. IEEE, 2023\.
- \[5\]Alexander R\. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev\.SummEval: Re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics, 9:568–600, 2021\.
- \[6\]Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald\.On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919\. Association for Computational Linguistics, 2020\.
- \[7\]Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher\.Evaluating the factual consistency of abstractive text summarization\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9332–9346\. Association for Computational Linguistics, 2020\.
- \[8\]Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang\.Entity\-level factual consistency of abstractive text summarization\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2727–2733\. Association for Computational Linguistics, 2021\.
- \[9\]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer\.BART: Denoising sequence\-to\-sequence pre\-training for natural language generation, translation, and comprehension\.arXiv preprint arXiv:1910\.13461, 2019\.
- \[10\]Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J\. Liu\.PEGASUS: Pre\-training with extracted gap\-sentences for abstractive summarization\.Proceedings of ICML 2020, 2020\.