Contrastive Attention Mechanism for Abstractive Sentence Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3044–3053, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3044 Contrastive Attention Mechanism for Abstractive Sentence Summarization Xiangyu Duan1,2∗, Hongfei Yu2∗, Mingming Yin2, Min Zhang1,2, Weihua Luo3, Yue Zhang4 1 Institute of Artificial Intelligence, Soochow University, Suzhou, China 2 School of Computer Science and Technology, Soochow University, Suzhou, China 3 Alibaba DAMO Academy, Hangzhou, China 4 School of Engineering, Westlake University, China xiangyuduan@suda.edu.cn; {hfyu,mmyin}@stu.suda.edu.cn; minzhang@suda.edu.cn weihua.luowh@alibaba-inc.com; yue.zhang@wias.org.cn Abstract We propose a contrastive attention mechanism to extend the sequence-to-sequence frame- work for abstractive sentence summarization task, which aims to generate a brief summary of a given source sentence. The proposed con- trastive attention mechanism accommodates two categories of attention: one is the conven- tional attention that attends to relevant parts of the source sentence, the other is the opponent attention that attends to irrelevant or less rel- evant parts of the source sentence. Both at- tentions are trained in an opposite way so that the contribution from the conventional atten- tion is encouraged and the contribution from the opponent attention is discouraged through a novel softmax and softmin functionality. Ex- periments on benchmark datasets show that, the proposed contrastive attention mechanism is more focused on the relevant parts for the summary than the conventional attention mechanism, and greatly advances the state- of-the-art performance on the abstractive sen- tence summarization task. We release the code at https://github.com/travel-go/ Abstractive-Text-Summarization. 1 Introduction Abstractive sentence summarization aims at gen- erating concise and informative summaries based on the core meaning of source sentences. Previ- ous endeavors tackle the problem through either rule-based methods (Dorr et al., 2003) or sta- tistical models trained on relatively small scale training corpora (Banko et al., 2000). Following its successful applications on machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), the sequence-to-sequence framework is also applied on the abstractive sentence summarization task us- ing large-scale sentence summary corpora (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., ∗ Equal contribution. 2016), obtaining better performance compared to the traditional methods. One central component in state-of-the-art se- quence to sequence models is the use of atten- tion for building connections between the source sequence and target words, so that a more in- formed decision can be made for generating a tar- get word by considering the most relevant parts of the source sequence (Bahdanau et al., 2015; Vaswani et al., 2017). For abstractive sentence summarization, such attention mechanisms can be useful for selecting the most salient words for a short summary, while filtering the negative influ- ence of redundant parts. We consider improving abstractive summariza- tion quality by enhancing target-to-source atten- tion. In particular, a contrastive mechanism is taken, by encouraging the contribution from the conventional attention that attends to relevant parts of the source sentence, while at the same time pe- nalizing the contribution from an opponent atten- tion that attends to …

Content truncated to 500 words