Abstractive Document Summarization with Summary-length Prediction

Findings of the Association for Computational Linguistics: EACL 2023, pages 618–624 May 2-6, 2023 ©2023 Association for Computational Linguistics Abstractive Document Summarization with Summary-length Prediction Jingun Kwon1, Hidetaka Kamigaito1,2, and Manabu Okumura1 1Tokyo Institute of Technology 2Nara Institute of Science and Technology (NAIST) kwon.j.ad@m.titech.ac.jp kamigaito.h@is.naist.jp oku@pi.titech.ac.jp Abstract Recently, we can obtain a practical abstrac- tive document summarization model by fine- tuning a pre-trained language model (PLM). Since the pre-training for PLMs does not con- sider summarization-specific information such as the target summary length, there is a gap between the pre-training and fine-tuning for PLMs in summarization tasks. To fill the gap, we propose a method for enabling the model to understand the summarization-specific infor- mation by predicting the summary length in the encoder and generating a summary of the predicted length in the decoder in fine-tuning. Experimental results on the WikiHow, NYT, and CNN/DM datasets showed that our meth- ods improve ROUGE scores from BART by generating summaries of appropriate lengths. Further, we observed about 3.0, 1,5, and 3.1 point improvements for ROUGE-1, -2, and - L, respectively, from GSum on the WikiHow dataset. Human evaluation results also showed that our methods improve the informativeness and conciseness of summaries. 1 Introduction Current abstractive summarization models mostly utilize pre-trained language models (PLMs) (Liu and Lapata, 2019; Dou et al., 2021; Liu and Liu, 2021; Narayan et al., 2021; Liu et al., 2022a). Ab- stractive document summarization requires an en- coder to determine the important parts in an input text and a decoder to output a non-redundant sum- mary of the appropriate length relevant to the input. Thus, the characteristics required for an abstractive summarization model differ from those required as a language model, and are not usually considered in the pre-training for PLMs (Devlin et al., 2019; Zhang et al., 2019; Lewis et al., 2020). Hence, we need to fine-tune a PLM with a summariza- tion dataset to treat it as an abstractive summariza- tion model. Unlike training a randomly initialized model, this fine-tuning maintains and inherits the Document Encoder Positional Embedding Layer Generating Summary Predicted Summary Length Predicted Summary Length Length Prediction Layer Subword Embedding Layer Decoder Length-Fusion Positional Encoding Layer Subword Embedding Layer Summary : Pretrained Layer : Randomly Initialized Layer Figure 1: Overview of our methods. The length pre- diction layer predicts the summary length. The length- fusion positional encoding layer controls the decoder to generate a summary of the appropriate summary length. parameters learned as an original language model. Therefore, to learn an abstractive summarization model by fine-tuning a PLM, it is necessary to sup- press its characteristics as a language model while enabling it to learn the unique properties of abstrac- tive summarization. For this purpose, we propose two regularization methods for fine-tuneing a PLM to learn abstrac- tive summarization. Figure 1 shows an overview of our methods. The first method is a regulariza- tion method that uses the encoder’s hidden states to predict the length of an output summary. When the length is not given …

Content truncated to 500 words