Length Control in Abstractive Summarization by Pretraining Information Selection

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6885 - 6895 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Length Control in Abstractive Summarization by Pretraining Information Selection Yizhu Liu1 Qi Jia2 Kenny Q. Zhu3∗ Shanghai Jiao Tong University Shanghai, China {1liuyizhu, 2Jia_qi}@sjtu.edu.cn, 3kzhu@cs.sjtu.edu.cn Abstract Previous length-controllable summarization models mostly control lengths at the decoding stage, whereas the encoding or the selection of information from the source document is not sensitive to the designed length. They also tend to generate summaries as long as those in the training data. In this paper, we propose a length-aware attention mechanism (LAAM) to adapt the encoding of the source based on the desired length. Our approach works by training LAAM on a summary length balanced dataset built from the original training data, and then fine-tuning as usual. Results show that this approach is effective in generating high-quality summaries with desired lengths and even those short lengths never seen in the original training set. 1 Introduction Abstractive summarization (Nallapati et al., 2016; See et al., 2017; Çelikyilmaz et al., 2018; Dong et al., 2019; Lewis et al., 2020; Liu et al., 2021; Dou et al., 2021) aims at reproducing the semantics and topics of the original text in a concise and fluent summary by paraphrasing. In order to display the summary on different mobile devices or websites with space limitations, we have to produce sum- maries in different lengths. Length-controllable summarization is a multi-objective optimization problem, including generating complete summaries within desired lengths and selecting proper infor- mation to summarize based on desired lengths. The existing length-controllable summarization based on encoder-decoder models can be divided into two categories: (1) early-stop during decoding and (2) information selection before encoding. Early-stop during decoding methods (Kikuchi et al., 2016; Liu et al., 2018; Makino et al., 2019; ∗Kenny Q. Zhu is the corresponding author, and is partially supported by NSFC Grant No. 91646205, and SJTU-CMBCC Joint Research Scheme. Source Document ... iranians erupted in celebration as young people waved flags from their sunroofs , blasted music from stereos and chatted online with the hashtag #irantalks . the excitement came after a breakthrough nuclear deal with the united states and other world powers ... Length Reference Summary 10 iranians celebrate the deal online and in the streets . 30 after a breakthrough nuclear agreement deal with the united states and other world powers , celebration broke out in iranians . young people waved flages and chatted online . Table 1: The reference summaries of one source docu- ment with lengths as 10 and 30. Yu et al., 2021) focus on when to output eos (end of sequence), indicating the end of the summary. An ad-hoc method (Rush et al., 2015) generates the eos by assigning a score of −∞ to all candidate words at the position of the desired length during test. Ad-hoc can be applied to any seq2seq model. Oth- ers learn the relationship between length and the …

Content truncated to 500 words