Concept Pointer Network for Abstractive Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3076–3085, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3076 Concept Pointer Network for Abstractive Summarization Wang Wenbo1, Gao Yang1,2∗ , Huang Heyan1,2 and Zhou Yuxiang1 1 School of Computer Science and Technology, Beijing Institute of Technology 2 Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications gyang@bit.edu.cn Abstract A quality abstractive summary should not only copy salient source texts as summaries but should also tend to generate new conceptual words to express concrete details. Inspired by the popular pointer generator sequence-to- sequence model, this paper presents a concept pointer network for improving these aspects of abstractive summarization. The network lever- ages knowledge-based, context-aware concep- tualizations to derive an extended set of candi- date concepts. The model then points to the most appropriate choice using both the con- cept set and original source text. This joint ap- proach generates abstractive summaries with higher-level semantic concepts. The train- ing model is also optimized in a way that adapts to different data, which is based on a novel method of distantly-supervised learning guided by reference summaries and testing set. Overall, the proposed approach provides sta- tistically significant improvements over sev- eral state-of-the-art models on both the DUC- 2004 and Gigaword datasets. A human evalu- ation of the model’s abstractive abilities also supports the quality of the summaries pro- duced within this framework. 1 Introduction Abstractive summarization (ABS) has gained overwhelming success owing to a tremendous development of sequence-to-sequence (seq2seq) model and its variants (Rush et al., 2015; Chopra et al., 2016; Paulus et al., 2017; Guo et al., 2018; Gao et al., 2019). In tandem with seq2seq mod- els, pointer generator was developed by See et al. (2017) as a solution to tackle the rare words and out-of-vocabulary (OOV) problem associated with generative-based models. The idea behind is to use attention as a pointer to determine the prob- ability of generating a word from both a vocab- ∗ Corresponding author Figure 1: “summary1” only copies keyword from the source text, while “summary2” generates new concepts to convey the meaning. ulary distribution and the source text. Pointer generator networks have also been extensively ac- cepted by the ABS community due to their effi- cacy with long document summaries (Chen and Bansal, 2018; Hsu et al., 2018), title summariza- tion (Sun et al., 2018), etc. However, the current power of abstractive sum- marization falls short of their potential. As the ex- ample in Figure 1 shows, a seq2seq model with a pointer mechanism (marked as the direct pointer) is likely to merely copy parts of the original text to form a summary using keywords and phrases, such as “317 athletes”. Conversely, a more human- like summary would be based on one’s own un- derstanding of the detail in the words, expressed as higher-level concepts drawn from world knowl- edge—like using the word “group” …

Content truncated to 500 words