Who did What: A Large-Scale Person-Centered Cloze Dataset

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Who did What: A Large-Scale Person-Centered Cloze Dataset Takeshi Onishi Hai Wang Mohit Bansal Kevin Gimpel David McAllester Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {tonishi,haiwang,mbansal,kgimpel,mcallester}@ttic.edu Abstract We have constructed a new “Who-did-What” dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension prob- lems constructed from the LDC English Gi- gaword newswire corpus. The WDW dataset has a variety of novel features. First, in con- trast with the CNN and Daily Mail datasets (Hermann et al., 2015) we avoid using article summaries for question formation. Instead, each problem is formed from two indepen- dent articles — an article given as the pas- sage to be read and a separate article on the same events used to form the question. Sec- ond, we avoid anonymization — each choice is a person named entity. Third, the problems have been filtered to remove a fraction that are easily solved by simple baselines, while re- maining 84% solvable by humans. We report performance benchmarks of standard systems and propose the WDW dataset as a challenge task for the community.1 1 Introduction Researchers distinguish the problem of general knowledge question answering from that of read- ing comprehension (Hermann et al., 2015; Hill et al., 2016). Reading comprehension is more diffi- cult than knowledge-based or IR-based question an- swering in two ways. First, reading comprehen- sion systems must infer answers from a given un- structured passage rather than structured knowledge sources such as Freebase (Bollacker et al., 2008) 1Available at tticnlp.github.io/who did what or the Google Knowledge Graph (Singhal, 2012). Second, machine comprehension systems cannot ex- ploit the large level of redundancy present on the web to find statements that provide a strong syntac- tic match to the question (Yang et al., 2015). In con- trast, a machine comprehension system must use the single phrasing in the given passage, which may be a poor syntactic match to the question. In this paper, we describe the construction of a new reading comprehension dataset that we refer to as “Who-did-What”. Two typical examples are shown in Table 1.2 The process of forming a prob- lem starts with the selection of a question article from the English Gigaword corpus. The question is formed by deleting a person named entity from the first sentence of the question article. An information retrieval system is then used to select a passage with high overlap with the first sentence of the question article, and an answer choice list is generated from the person named entities in the passage. Our dataset differs from the CNN and Daily Mail comprehension tasks (Hermann et al., 2015) in that it forms questions from two distinct articles rather than summary points. This allows problems to be derived from document collections that do not con- tain manually-written summaries. This also reduces the syntactic similarity between the question and the relevant sentences in …

Content truncated to 500 words