Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5021–5027 August 1–6, 2021. ©2021 Association for Computational Linguistics 5021 Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization Shuaiqi LIU, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen Department of Computing, The Hong Kong Polytechnic University {cssqliu, csjcao, csryang, cszwen}@comp.polyu.edu.hk Abstract Abstractive multi-document summarization aims to generate a comprehensive summary covering salient content from multiple input documents. Compared with previous RNN- based models, the Transformer-based models employ the self-attention mechanism to cap- ture the dependencies in input documents and can generate better summaries. Existing works have not considered key phrases in determin- ing attention weights of self-attention. Conse- quently, some of the tokens within key phrases only receive small attention weights. It can affect completely encoding key phrases that convey the salient ideas of input documents. In this paper, we introduce the Highlight- Transformer, a model with the highlighting mechanism in the encoder to assign greater attention weights for the tokens within key phrases. We propose two structures of high- lighting attention for each head and the multi- head highlighting attention. The experimental results on the Multi-News dataset show that our proposed model significantly outperforms the competitive baseline models. 1 Introduction Abstractive Multi-Document Summarization (MDS) offers the challenge of generating a comprehensive summary of multiple related documents. It requires summarization models to capture the salient content from input documents. Compared with the previous RNN-based models for abstractive MDS, the Transformer-based models (Gehrmann et al., 2018; Liu et al., 2018; Liu and Lapata, 2019a; Li et al., 2020b) employ the self-attention mechanism to capture the dependencies in input documents, and they can generate better summaries. Calculating attention weights is a crucial step in the self-attention mechanism. Input documents usually contain some key phrases that convey the Highlighting Matrix Self-Attention Weight Matrix Highlighting Attention Weight Matrix Figure 1: The highlighting mechanism assigns greater attention weights for tokens within key phrases indi- cated by the highlighting matrix. salient ideas of input documents. However, exist- ing works have not considered key phrases in de- termining attention weights of self-attention. Key phrases usually comprise multiple tokens, which should be highly related and serve as a complete grammatical unit in input documents. When testing Transformer-based models, we observe some of the tokens within key phrases only receive small atten- tion weights, which can affect completely encoding key phrases and the salient ideas they convey. In this paper, we propose the Highlight- Transformer, an abstractive summarization model with the highlighting mechanism in the encoder. As depicted in Figure 1, the highlighting mecha- nism assigns greater attention weights for tokens within key phrases. Furthermore, the highlighting mechanism mainly comprises three parts: the high- lighting matrix, the highlighting attention, and the multi-head highlighting attention. 5022 Our work is inspired by previous studies in ed- ucation and psychology that indicate key phrases are important for people to understand (Rello et al., 2014; Hargreaves and Crabb, 2016) and summarize (Benzer et al., 2016; Chou, 2012) the given docu- ments. …

Content truncated to 500 words