Constructing Datasets for Multi-hop Reading Comprehension Across Documents

Constructing Datasets for Multi-hop Reading Comprehension Across Documents Johannes Welbl1 Pontus Stenetorp1 Sebastian Riedel1,2 1University College London, 2Bloomsbury AI {j.welbl,p.stenetorp,s.riedel}@cs.ucl.ac.uk Abstract Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or docu- ment. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently no resources exist to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and com- bine evidence – effectively performing multi- hop, alias multi-step, inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced,1 and we identify potential pitfalls and devise circum- vention strategies. We evaluate two previ- ously proposed competitive models and find that one can integrate information across doc- uments. However, both models struggle to se- lect relevant information; and providing doc- uments guaranteed to be relevant greatly im- proves their performance. While the mod- els outperform several strong baselines, their best accuracy reaches 54.5% on an annotated test set, compared to human performance at 85.0%, leaving ample room for improvement. 1 Introduction Devising computer systems capable of answering questions about knowledge described using text has 1Available at http://qangaroo.cs.ucl.ac.uk The Hanging Gardens, in [Mumbai], also known as Pherozeshah Mehta Gardens, are terraced gardens … They provide sunset views over the [Arabian Sea] … Mumbai (also known as Bombay, the official name until 1995) is the capital city of the Indian state of Maharashtra. It is the most populous city in India … Q: (Hanging gardens of Mumbai, country, ?) Options: {Iran, India, Pakistan, Somalia, …} The Arabian Sea is a region of the northern Indian Ocean bounded on the north by Pakistan and Iran, on the west by northeastern Somalia and the Arabian Peninsula, and on the east by India … Figure 1: A sample from the WIKIHOP dataset where it is necessary to combine information spread across multi- ple documents to infer the correct answer. been a longstanding challenge in Natural Language Processing (NLP). Contemporary end-to-end Read- ing Comprehension (RC) methods can learn to ex- tract the correct answer span within a given text and approach human-level performance (Kadlec et al., 2016; Seo et al., 2017a). However, for exist- ing datasets, relevant information is often concen- trated locally within a single sentence, emphasizing the role of locating, matching, and aligning informa- tion between query and support text. For example, Weissenborn et al. (2017) observed that a simple bi- nary word-in-query indicator feature boosted the rel- ative accuracy of a baseline model by 27.9%. We argue that, in order to further the ability of ma- chine comprehension methods to extract knowledge from text, we must move beyond a scenario where relevant information is coherently and explicitly stated within a single document. …

Content truncated to 500 words