Searching for Effective Neural Extractive Summarization: What Works and What’s Next

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049–1058 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1049 Searching for Effective Neural Extractive Summarization: What Works and What’s Next Ming Zhong∗, Pengfei Liu∗, Danqing Wang, Xipeng Qiu†, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {mzhong18,pfliu14,dqwang18,xpqiu,xjhuang}@fudan.edu.cn Abstract The recent years have seen remarkable suc- cess in the use of deep neural networks on text summarization. However, there is no clear un- derstanding of why they perform so well, or how they might be improved. In this paper, we seek to better understand how neural extractive summarization systems could benefit from dif- ferent types of model architectures, transfer- able knowledge and learning schemas. Addi- tionally, we find an effective way to improve current frameworks and achieve the state-of- the-art result on CNN/DailyMail by a large margin based on our observations and analy- ses. Hopefully, our work could provide more clues for future research on extractive sum- marization. Source code will be available on Github1. 1 Introduction Recent years has seen remarkable success in the use of deep neural networks for text summariza- tion (See et al., 2017; Celikyilmaz et al., 2018; Jadhav and Rajan, 2018). So far, most research utilizing the neural network for text summariza- tion has revolved around architecture engineer- ing (Zhou et al., 2018; Chen and Bansal, 2018; Gehrmann et al., 2018). Despite their success, it remains poorly under- stood why they perform well and what their short- comings are, which limits our ability to design bet- ter architectures. The rapid development of neural architectures calls for a detailed empirical study of analyzing and understanding existing models. In this paper, we primarily focus on extractive summarization since they are computationally ef- ficient, and can generate grammatically and coher- ent summaries (Nallapati et al., 2017). and seek to ∗These two authors contributed equally. †Corresponding author. 1https://github.com/fastnlp/fastNLP better understand how neural network-based ap- proaches to this task could benefit from different types of model architectures, transferable knowl- edge, and learning schemas, and how they might be improved. Architectures Architecturally, the better perfor- mance usually comes at the cost of our under- standing of the system. To date, we know little about the functionality of each neural component and the differences between them (Peters et al., 2018b), which raises the following typical ques- tions: 1) How does the choice of different neu- ral architectures (CNN, RNN, Transformer) influ- ence the performance of the summarization sys- tem? 2) Which part of components matters for specific dataset? 3) Do current models suffer from the over-engineering problem? Understanding the above questions can not only help us to choose suitable architectures in different application scenarios, but motivate us to move for- ward to more powerful frameworks. External Transferable Knowledge and Learn- ing schemas Clearly, the improvement in accu- racy and performance is not merely because of the shift from feature engineering …

Content truncated to 500 words