Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa
Summary
A comparative study of BART, BERT, and RoBERTa for text summarization, examining their architectures and suitability for extractive and abstractive summarization tasks.
View Cached Full Text
Cached at: 08/21/26, 09:56 AM
# Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa Source: [https://arxiv.org/abs/2608.19200](https://arxiv.org/abs/2608.19200) [View PDF](https://arxiv.org/pdf/2608.19200) > Abstract:Text summarization refers to the task of condensing a document into a shorter version while preserving its key information\. Automatic text summarization \(ATS\), driven by advancements in natural language processing \(NLP\), has developed rapidly in recent years\. ATS methods are commonly categorized by input type \(such as single\-document or multi\-document summarization\) and by output type \(extractive, abstractive, and hybrid\)\. This article presents a focused review of modern summarization techniques with an emphasis on transformer based models and large language models \(LLMs\), specifically BERT, RoBERTa and BART\. It examines their architectures, pretraining strategies, and their suitability for extractive and abstractive summarization tasks\. ## Submission history From: Vinayak Elangovan \[[view email](https://arxiv.org/show-email/a408dfef/2608.19200)\] **\[v1\]**Tue, 2 Jun 2026 19:49:08 UTC \(333 KB\)
Similar Articles
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
This paper introduces Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) metrics to quantify abstractiveness in text summarization, using harmonic mean of document lengths and cubic non-overlap factor. Empirical evaluation on XSUM with four models shows the metrics effectively discriminate between extractive and abstractive summaries, and flag potential hallucination.
MASF: A Multi-Model Adaptive Selection Framework for Abstractive Text summarization
Presents MASF, a multi-model adaptive selection framework that integrates multiple fine-tuned transformer summarization models and selects the highest-quality summary, achieving 88.63% BERTScore on CNN/DailyMail and outperforming several LLMs.
A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses
This paper compares Structural Topic Models (STM) and BERTopic for analyzing short, open-ended survey responses, finding that BERTopic with contextual augmentation yields better topic coherence and interpretability, while STM offers stronger support for inferential covariate analysis.
COBART: Controlled, Optimized, Bidirectional and Auto-Regressive Transformer for Ad Headline Generation
This paper presents COBART, a method that fine-tunes BART with prefix control tokens to generate ad headlines with controllable length and optimized click-through rate (CTR), achieving a 25.82% improvement in ROUGE-L and 5.82% in estimated CTR over previous baselines.
Optimizing Abstractive Summarization With Fine-Tuned PEGASUS
This paper presents fine-tuning of PEGASUS on the XL-Sum English corpus, achieving state-of-the-art results with significant improvements over the baseline mT5 model across ROUGE scores.