Accurate Supervised and Semi-Supervised Machine Reading for Long Documents

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2011–2020 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Accurate Supervised and Semi-Supervised Machine Reading for Long Documents Izzeddin Gur∗ Department of Computer Science University of California, Santa Barbara izzeddingur@cs.ucsb.edu Daniel Hewlett Google Research dhewlett@google.com Alexandre Lacoste† Element AI alex.lacoste.shmu@gmail.com Llion Jones Google Research llion@google.com Abstract We introduce a hierarchical architecture for machine reading capable of extract- ing precise information from long doc- uments. The model divides the docu- ment into small, overlapping windows and encodes all windows in parallel with an RNN. It then attends over these window encodings, reducing them to a single en- coding, which is decoded into an answer using a sequence decoder. This hierarchi- cal approach allows the model to scale to longer documents without increasing the number of sequential steps. In a super- vised setting, our model achieves state of the art accuracy of 76.8 on the WikiRead- ing dataset. We also evaluate the model in a semi-supervised setting by downsam- pling the WikiReading training set to cre- ate increasingly smaller amounts of su- pervision, while leaving the full unlabeled document corpus to train a sequence au- toencoder on document windows. We evaluate models that can reuse autoen- coder states and outputs without fine- tuning their weights, allowing for more ef- ficient training and inference. 1 Introduction Recently, deep neural networks (DNNs) have pro- vided promising results for a variety of reading comprehension and question answering tasks (We- ston et al., 2014; Hermann et al., 2015; Rajpurkar et al., 2016), which require extracting precise in- formation from documents conditioned on a query. While a basic sequence to sequence (seq2seq) model (Sutskever et al., 2014) can perform these ∗Work completed while interning at Google Research. †Work completed while at Google Research. tasks by encoding a question and document se- quence and decoding an answer sequence (Hewlett et al., 2016), it has some disadvantages. The an- swer may be encountered early in the text and need to be stored across all the further recurrent steps, leading to forgetting or corruption; Atten- tion can be added to the decoder to solve this problem (Hermann et al., 2015). Even with at- tention, approaches based on Recurrent Neural Networks (RNNs) require a number of sequential steps proportional to the document length to en- code each document position. Hierarchical read- ing models address this problem by breaking the document into sentences (Choi et al., 2017). In this paper, we introduce a simpler hierarchical model that achieves state-of-the-art performance on our benchmark task without this linguistic structure, and use it as framework to explore semi- supervised learning for reading comprehension. We first develop a hierarchical reader called Sliding-Window Encoder Attentive Reader (SWEAR) that circumvents the aforementioned bottlenecks of existing readers. SWEAR, illus- trated in Figure 1, first encodes each question into a vector space representation. It then chunks each document into overlapping, fixed-length windows and, conditioned on the question representation, encodes each window in parallel. Inspired …

Content truncated to 500 words