Self-Attention Guided Copy Mechanism for Abstractive Summarization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1355–1362 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1355 Self-Attention Guided Copy Mechanism for Abstractive Summarization Song Xu∗, Haoran Li∗, Peng Yuan, Youzheng Wu, Xiaodong He and Bowen Zhou JD AI Research {xusong28,lihaoran24,yuanpeng29,wuyouzheng1}@jd.com {xiaodong.he,bowen.zhou}@jd.com Abstract Copy module has been widely equipped in the recent abstractive summarization models, which facilitates the decoder to extract words from the source into the summary. Generally, the encoder-decoder attention is served as the copy distribution, while how to guarantee that important words in the source are copied re- mains a challenge. In this work, we propose a Transformer-based model to enhance the copy mechanism. Specifically, we identify the im- portance of each source word based on the de- gree centrality with a directed graph built by the self-attention layer in the Transformer. We use the centrality of each source word to guide the copy process explicitly. Experimental re- sults show that the self-attention graph pro- vides useful guidance for the copy distribution. Our proposed models significantly outperform the baseline methods on the CNN/Daily Mail dataset and the Gigaword dataset. 1 Introduction The explosion of information has expedited the rapid development of text summarization technol- ogy, which can help us to grasp the key points from miscellaneous information quickly. There are broadly two types of summarization methods: extractive and abstractive. Extractive approaches select the original text segments in the input to form a summary, while abstractive approaches “create” novel sentences based on natural language genera- tion techniques. In the past few years, recurrent neural networks (RNNs) based architectures (Chopra et al., 2016; Gu et al., 2016; Nallapati et al., 2016, 2017; See et al., 2017; Zhou et al., 2017; Li et al., 2018b,a; Zhu et al., 2019) have obtained state-of-the-art re- sults for text summarization. Benefit from long- term dependency and high scalability, transformer- based networks have shown superiority over RNNs ∗Equal contribution. Source: two u.s. senators are blocking 11 of president barack obama ’s nominees for senior administration posts at the pentagon and justice department in protest over a proposal to house guantanamo detainees at the fort leavenworth prison in their midwestern home state of kansas Reference: us senators bar obama nominees protest guantanamo Transformer: 1 us senators block pentago justice nominees Transformer + Copy: us senators block 11 from pentago justice posts Transformer + Guided Copy: us senators block obama nominees over guantanamo Top Words from Self-attention: nominees, obama, senators, pentagon, guantanamo Table 1: Yellow shades represent overlap with ref- erence. The above summary generated by standard copy mechanism miss some importance words, such as “obama” and “nominees”. on many NLP tasks, including machine transla- tion (Vaswani et al., 2017; Dehghani et al., 2019), sentence classification (Devlin et al., 2019; Cohan et al., 2019), and text summarization (Song et al., 2019; Zhang et al., 2019). One of the most successful frameworks for the summarization task is Pointer-Generator Net- work (See et al., 2017) that combines extractive and abstractive …

Content truncated to 500 words