Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 704–717 June 6–11, 2021. ©2021 Association for Computational Linguistics 704 Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation Alexander R. Fabbri† Simeng Han†† Haoyuan Li‡‡ Haoran Li ‡ Marjan Ghazvininejad‡ Shafiq Joty†† Dragomir Radev† Yashar Mehdad‡ † Yale University ‡ Facebook AI †† Nanyang Technological University ‡‡ Renmin University of China {alexander.fabbri,dragomir.radev}@yale.edu {aimeeli,ghazvini,mehdad}@fb.com Abstract Models pretrained with self-supervised objec- tives on large text corpora achieve state-of-the- art performance on English text summariza- tion tasks. However, these models are typi- cally fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche do- mains. In this work, we introduce a novel and generalizable method, called WikiTrans- fer, for fine-tuning pretrained models for sum- marization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain charac- teristics of the target dataset, such as the length and level of abstraction of the desired sum- maries. WikiTransfer models achieve state-of- the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our ap- proach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training exam- ples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset as- pects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsu- pervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation. 1 Introduction Automatic text summarization aims to distill the most salient content of a given text in a compact form. Recent advances in summarization have been driven by the availability of large-scale datasets such as the CNN-DailyMail (CNNDM) corpus (Nallapati et al., 2016) and the New York Times corpus (Sandhaus, 2008) as well as by the intro- duction of large pretrained models such as BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2019), in some cases resulting in summaries which are even favored over the human-written reference summaries. Creating data for every new domain, however, is infeasible and highly costly. Thus, the ability to transfer large pretrained models to new domains with little or no in-domain data is neces- sary, especially as such models make their way into real-world applications. Unsupervised summarization approaches in- clude autoencoders to mirror the information com- pression inherent in summarization (Baziotis et al., 2019; Chu and Liu, 2019; Bražinskas et al., 2020b) as well as large-scale pretraining for domain- specific adaptation (Yang et al., 2020). However, little work has focused on domain adaptation in summarization. Wang et al. (2019) examine do- main adaptation for extractive summarization. Hua and Wang (2017) …

Content truncated to 500 words