Attention Optimization for Abstractive Document Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1222–1228, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1222 Attention Optimization for Abstractive Document Summarization Min Gui1, Junfeng Tian1, Rui Wang1, Zhenglu Yang2 1 Alibaba Group, China 2 School of Computer Science, Nankai University, China {guimin.gm, tjf141457, masi.wr}@alibaba-inc.com yangzl@nankai.edu.cn Abstract Attention plays a key role in the improve- ment of sequence-to-sequence-based docu- ment summarization models. To obtain a pow- erful attention helping with reproducing the most salient information and avoiding repeti- tions, we augment the vanilla attention model from both local and global aspects. We pro- pose an attention refinement unit paired with local variance loss to impose supervision on the attention model at each decoding step, and a global variance loss to optimize the atten- tion distributions of all decoding steps from the global perspective. The performances on the CNN/Daily Mail dataset verify the effec- tiveness of our methods. 1 Introduction Abstractive document summarization (Rush et al., 2015; Nallapati et al., 2016; Tan et al., 2017; Chen and Bansal, 2018; Celikyilmaz et al., 2018) at- tempts to produce a condensed representation of the most salient information of the document, as- pects of which may not appear as parts of the original input text. One popular framework used in abstractive summarization is the sequence-to- sequence model introduced by Sutskever et al. (2014). The attention mechanism (Bahdanau et al., 2014) is proposed to enhance the sequence- to-sequence model by allowing salient features to dynamically come to the forefront as needed to make up for the incapability of memorizing the long input source. However, when it comes to longer documents, basic attention mechanism may lead to distraction and fail to attend to the relatively salient parts. Therefore, some works focus on designing vari- ous attentions to tackle this issue (Tan et al., 2017; Gehrmann et al., 2018). We follow this line of re- search and propose an effective attention refine- ment unit (ARU). Consider the following case. Even with a preliminary idea of which parts of source document should be focused on (attention), sometimes people may still have trouble in decid- ing which exact part should be emphasized for the next word (the output of the decoder). To make a more correct decision on what to write next, people always adjust the concentrated content by reconsidering the current state of what has been summarized already. Thus, ARU is designed as an update unit based on current decoding state, aiming to retain the attention on salient parts but weaken the attention on irrelevant parts of input. The de facto standard attention mechanism is a soft attention that assigns attention weights to all input encoder states, while according to previ- ous work (Xu et al., 2015; Shankar et al., 2018), a well-trained hard attention on exact one input state is conducive to more accurate results compared to the soft attention. To maintain good performance of hard attention …

Content truncated to 500 words