Reading Like HER: Human Reading Inspired Extractive Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3033–3043, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3033 Reading Like HER: Human Reading Inspired Extractive Summarization Ling Luo1,3, Xiang Ao1,3∗, Yan Song2, Feiyang Pan1,3, Min Yang4, Qing He1,3 1Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China 2Sinovation Ventures 3University of Chinese Academy of Sciences 4Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences {luoling18s, aoxiang, panfeiyang, heqing}@ict.ac.cn, clksong@gmail.com Abstract In this work, we re-examine the problem of extractive text summarization for long docu- ments. We observe that the process of extract- ing summarization of human can be divided into two stages: 1) a rough reading stage to look for sketched information, and 2) a sub- sequent careful reading stage to select key sentences to form the summary. By simulat- ing such a two-stage process, we propose a novel approach for extractive summarization. We formulate the problem as a contextual- bandit problem and solve it with policy gra- dient. We adopt a convolutional neural net- work to encode gist of paragraphs for rough reading, and a decision making policy with an adapted termination mechanism for careful reading. Experiments on the CNN and Daily- Mail datasets show that our proposed method can provide high-quality summaries with var- ied length, and significantly outperform the state-of-the-art extractive methods in terms of ROUGE metrics. 1 Introduction Automatic text summarization has wide popular- ity in NLP applications such as producing di- gests, headlines and reports. Among the super- vised methods, two main types are usually ex- plored, namely abstractive and extractive sum- marizations (Nenkova et al., 2011). Compared with abstractive approaches, extractive methods are more practical and applicable as they are faster, simpler and more reliable on grammar as well as semantic information (Yao et al., 2018). Recent studies (Cheng and Lapata, 2016; Nal- lapati et al., 2017; Yasunaga et al., 2017; Feng et al., 2018) consider extractive summarization as a sequence labeling task, where each sen- tence is individually processed and determined ∗Corresponding author. whether it should be extracted or not. Vari- ous neural networks are used to label each sen- tence and trained using cross-entropy loss to max- imize the likelihood of the ground-truth labeled sequences, which may derive the mismatch be- tween the cross-entropy objective function and the evaluation criterion. On the other hand, some reinforcement learning based methods (Wu and Hu, 2018; Narayan et al., 2018; Yao et al., 2018) directly optimize the evaluation metric by com- bining cross-entropy loss with rewards and train model with policy gradient reinforcement learn- ing. Note that the rewards usually reflect the qual- ity of extracted summary and measured by stan- dard evaluation protocol. However, they still se- quentially process text and tend to extract earlier sentences over later ones due to the sequential na- ture of selection (Dong et al., 2018). Although great efforts have been devoted to this …

Content truncated to 500 words