Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1108–1118, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers Anupam Guha,1 Mohit Iyyer,1 Danny Bouman,1 Jordan Boyd-Graber2 1University of Maryland, Department of Computer Science and umiacs 2University of Colorado, Department of Computer Science aguha@cs.umd.edu, miyyer@umiacs.umd.edu, dannybb@gmail.com, Jordan.Boyd.Graber@colorado.edu Abstract Coreference is a core nlp problem. How- ever, newswire data, the primary source of existing coreference data, lack the rich- ness necessary to truly solve coreference. We present a new domain with denser references—quiz bowl questions—that is challenging and enjoyable to humans, and we use the quiz bowl community to develop a new coreference dataset, together with an annotation framework that can tag any text data with coreferences and named entities. We also successfully integrate active learn- ing into this annotation pipeline to collect documents maximally useful to coreference models. State-of-the-art coreference sys- tems underperform a simple classifier on our new dataset, motivating non-newswire data for future coreference research. 1 Introduction Coreference resolution—adding annotations to an input text where multiple strings refer to the same entity—is a fundamental problem in com- putational linguistics. It is challenging because it requires the application of syntactic, semantic, and world knowledge (Ng, 2010). For example, in the sentence Monsieur Poirot assured Hastings that he ought to have faith in him, the strings Monsieur Poirot and him refer to the same person, while Hastings and he refer to a different character. There are a panoply of sophisticated corefer- ence systems, both data-driven (Fernandes et al., 2012; Durrett and Klein, 2013; Durrett and Klein, 2014; Bj¨orkelund and Kuhn, 2014) and rule-based (Pradhan et al., 2011; Lee et al., 2011). Recent CoNLL shared tasks provide the oppor- tunity to make a fair comparison between these systems. However, because all of these shared tasks contain strictly newswire data,1 it is unclear how existing systems perform on more diverse data. We argue in Section 2 that to truly solve coref- erence resolution, the research community needs high-quality datasets that contain many chal- lenging cases such as nested coreferences and coreferences that can only be resolved using ex- ternal knowledge. In contrast, newswire is delib- erately written to contain few coreferences, and those coreferences should be easy for the reader to resolve. Thus, systems that are trained on such data commonly fail to detect coreferences in more expressive, non-newswire text. Given newswire’s imperfect range of corefer- ence examples, can we do better? In Section 3 we present a specialized dataset that specifically tests a human’s coreference resolution ability. This dataset comes from a community of trivia fans who also serve as enthusiastic annotators (Section 4). These data have denser coreference mentions than newswire text and present hith- erto unexplored questions of what is coreferent and what is not. We also incorporate active learn- ing into the annotation process. The result is a small but highly dense dataset of …

Content truncated to 500 words