Learning to Skim Text

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1880–1890 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1172 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1880–1890 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1172 Learning to Skim Text Adams Wei Yu∗ Carnegie Mellon University weiyu@cs.cmu.edu Hongrae Lee Google hrlee@google.com Quoc V. Le Google qvl@google.com Abstract Recurrent Neural Networks are showing much promise in many sub-areas of nat- ural language processing, ranging from document classification to machine trans- lation to automatic question answering. Despite their promise, many recurrent models have to read the whole text word by word, making it slow to handle long documents. For example, it is difficult to use a recurrent network to read a book and answer questions about it. In this paper, we present an approach of read- ing text while skipping irrelevant informa- tion if needed. The underlying model is a recurrent network that learns how far to jump after reading a few words of the input text. We employ a standard policy gradient method to train the model to make discrete jumping decisions. In our benchmarks on four different tasks, including number prediction, sentiment analysis, news arti- cle classification and automatic Q&A, our proposed model, a modified LSTM with jumping, is up to 6 times faster than the standard sequential LSTM, while main- taining the same or even better accuracy. 1 Introduction The last few years have seen much success of ap- plying neural networks to many important appli- cations in natural language processing, e.g., part- of-speech tagging, chunking, named entity recog- nition (Collobert et al., 2011), sentiment analy- sis (Socher et al., 2011, 2013), document classifi- cation (Kim, 2014; Le and Mikolov, 2014; Zhang et al., 2015; Dai and Le, 2015), machine transla- tion (Kalchbrenner and Blunsom, 2013; Sutskever ∗Most of work was done when AWY was with Google. et al., 2014; Bahdanau et al., 2014; Sennrich et al., 2015; Wu et al., 2016), conversational/dialogue modeling (Sordoni et al., 2015; Vinyals and Le, 2015; Shang et al., 2015), document summariza- tion (Rush et al., 2015; Nallapati et al., 2016), parsing (Andor et al., 2016) and automatic ques- tion answering (Q&A) (Weston et al., 2015; Her- mann et al., 2015; Wang and Jiang, 2016; Wang et al., 2016; Trischler et al., 2016; Lee et al., 2016; Seo et al., 2016; Xiong et al., 2016). An important characteristic of all these models is that they read all the text available to them. While it is essential for certain applications, such as machine transla- tion, this characteristic also makes it slow to ap- ply these models to scenarios that have long input text, such as document classification or automatic Q&A. However, the fact that texts are usually writ- ten with redundancy inspires us to think about the possibility of reading selectively. In this paper, we consider the problem of under- standing documents with partial …

Content truncated to 500 words