AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5892–5904 June 6–11, 2021. ©2021 Association for Computational Linguistics 5892 AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization Tiezheng Yu∗, Zihan Liu∗, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {tyuah,zliucr}@connect.ust.hk, pascale@ece.ust.hk Abstract State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three dif- ferent settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task- adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model’s catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.1 1 Introduction Abstractive summarization models aim to extract essential information from long documents and to generate short, concise and readable text. Re- cently, neural abstractive summarization models have achieved remarkable performance (Gehrmann et al., 2018; Paulus et al., 2018), and large-scale generative pre-training (Lewis et al., 2019; Raffel et al., 2019) has shown itself to be surprisingly effective at generation tasks, including abstrac- tive summarization. However, these models gen- erally require large numbers of human-annotated summaries to achieve state-of-the-art performance, ∗ Equal contributions. Listing order is random. 1The code and data are released at: https://github. com/TysonYu/AdaptSum which makes them not scalable to low-resource do- mains where only a few labeled data are available. Domain adaptation methods have naturally arisen to tackle the low-resource issue and enable models to quickly adapt to target domain tasks. Yet, despite their practicality, very few studies have used domain adaptation methods on the low- resource scenario for the abstractive summariza- tion task. To address this research gap, we present AdaptSum, the first benchmark to simulate the low-resource domain Adaptation setting for ab- stractive Summarization systems with a combina- tion of existing datasets across six diverse domains (dialog (Gliwa et al., 2019), email (Zhang and Tetreault, 2019), movie review (Wang and Ling, 2016), debate (Wang and Ling, 2016), social me- dia (Kim et al., 2019), and science (Yasunaga et al., 2019)), and for each domain, we reduce the num- ber of training samples to a small quantity so as to create a low-resource scenario. Recently, conducting a second pre-training step on large-scale language models (e.g., BERT (De- vlin et al., 2019), RoBERTa (Liu et al., 2019a)) has proven to be effective for …

Content truncated to 500 words