Pre-training for Abstractive Document Summarization by Reinstating Source Text

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3646–3660, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3646 Pre-training for Abstractive Document Summarization by Reinstating Source Text Yanyan Zou1,3∗, Xingxing Zhang2, Wei Lu3, Furu Wei2 and Ming Zhou2 1 JD.com 2 Microsoft Research Asia, Beijing, China 3 StatNLP Research Group, Singapore University of Technology and Design zoe.yyzou@gmail.com {xizhang,fuwei,mingzhou}@microsoft.com luwei@sutd.edu.sg Abstract Abstractive document summarization is usu- ally modeled as a sequence-to-sequence (SEQ2SEQ) learning problem. Unfortunately, training large SEQ2SEQ based summarization models on limited supervised summarization data is challenging. This paper presents three sequence-to-sequence pre-training (in short- hand, STEP) objectives which allow us to pre-train a SEQ2SEQ based abstractive sum- marization model on unlabeled text. The main idea is that, given an input text artifi- cially constructed from a document, a model is pre-trained to reinstate the original docu- ment. These objectives include sentence re- ordering, next sentence generation and masked document generation, which have close rela- tions with the abstractive document summa- rization task. Experiments on two benchmark summarization datasets (i.e., CNN/DailyMail and New York Times) show that all three ob- jectives can improve performance upon base- lines. Compared to models pre-trained on large-scale data (≥160GB), our method, with only 19GB text for pre-training, achieves comparable results, which demonstrates its effectiveness. Code and models are pub- lic available at https://github.com/ zoezou2015/abs_pretraining. 1 Introduction Automatic document summarization is the task of condensing a document into its shorter form with important content preserved, which requires wide- coverage understandings of the document, rather than specific words or phrases. This task can be typically classified into two categories: extractive and abstractive document summarization. Extrac- tive summarization (Cheng and Lapata, 2016; Nal- lapati et al., 2017; Narayan et al., 2018a) aims to ∗ Contribution during internship at Microsoft Research. The first author now works in JD.com. extract important sentences from the input docu- ment and concatenates such extracted sentences as the corresponding output summary. Thus, the rel- ative orders of the selected sentences in the sum- mary is the same as their relative orders in the in- put document. Differently, abstractive summariza- tion (Nallapati et al., 2016; See et al., 2017; Paulus et al., 2018) rewrites the source text and gener- ates the corresponding summary which may con- tain novel words and phrases not featured in the in- put. The output summary is closely related to the input document. Also, summary sentences, para- phrased from the input by the abstractive summa- rizers, might have a different relative order com- pared to the source text. In other words, contents of the original document may be reordered in its summary. Such a phenomena is defined as con- tent reordering (see Section 3.2 for detailed defi- nition). Statistically, we observed that around 40% instances of the training split of our summarization dataset have this content reordering phenomena. Therefore, it is necessary to design a model that is capable of reordering content. However, as far as we know, relatively rare prior work has studied this …

Content truncated to 500 words