Learning to Search in Long Documents Using Document Structure

Proceedings of the 27th International Conference on Computational Linguistics, pages 161–176 Santa Fe, New Mexico, USA, August 20-26, 2018. 161 Learning to Search in Long Documents Using Document Structure Mor Geva Tel Aviv University morgeva@mail.tau.ac.il Jonathan Berant Tel Aviv University joberant@cs.tau.ac.il Abstract Reading comprehension models are based on recurrent neural networks that sequentially process the document tokens. As interest turns to answering more complex questions over longer docu- ments, sequential reading of large portions of text becomes a substantial bottleneck. Inspired by how humans use document structure, we propose a novel framework for reading comprehension. We represent documents as trees, and model an agent that learns to interleave quick navigation through the document tree with more expensive answer extraction. To encourage exploration of the document tree, we propose a new algorithm, based on Deep Q-Network (DQN), which strate- gically samples tree nodes at training time. Empirically we find our algorithm improves question answering performance compared to DQN and a strong information-retrieval (IR) baseline, and that ensembling our model with the IR baseline results in further gains in performance. 1 Introduction Reading comprehension (RC), the task of reading documents and answering questions about their con- tent, has attracted immense attention recently. While early work focused on simple questions and short paragraphs (Hermann et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2017; Onishi et al., 2016), current work is shifting towards more complex questions that require reasoning over long documents (Joshi et al., 2017; Hewlett et al., 2016; Welbl et al., 2017; Koˇcisky et al., 2017). Long documents pose a challenge for current RC models, as they are dominated by recurrent neural networks (RNNs) (Chen et al., 2016; Kadlec et al., 2016; Xiong et al., 2017). RNNs process documents token-by-token, and thus using them for long documents is prohibitive. A common solution is to retrieve part of the document with an IR approach (Chen et al., 2017; Clark and Gardner, 2017) or a cheap model (Watanabe et al., 2017), and run an RNN over the retrieved excerpts. However, as documents become longer and questions become complex, two problems emerge: (a) retrieving all the necessary text with a one-shot IR method when performing complex reasoning becomes harder, and thus thousands of tokens are retrieved (Clark and Gardner, 2017). (b) Running even a cheap model over the document in its entirety becomes expensive (Choi et al., 2017). Humans, in lieu of a mental inverted index, use document structure to guide their search for answers. E.g., the answer to “What high school did Leonard Cohen go to?” is likely to appear in “Early life”, while the answer to “How hot is it in Melbourne in July?” is likely to appear in “Climate”. In this work we investigate whether we can train a model to navigate through a document using its structure and find the answer, while reading only a small portion of the entire document. We represent documents as trees and train an agent that navigates through the document tree until returning a …

Content truncated to 500 words