WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4303–4312, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4303 WikiCREM: A Large Unsupervised Corpus for Coreference Resolution Vid Kocijan1 Oana-Maria Camburu1,2 Ana-Maria Cret¸u3 Yordan Yordanov1 Phil Blunsom1,4 Thomas Lukasiewicz1,2 1University of Oxford, UK 2Alan Turing Institute, London, UK 3Imperial College London, UK 4DeepMind, London, UK firstname.lastname@cs.ox.ac.uk, a.cretu@imperial.ac.uk Abstract Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually la- belling data is costly. In this work, we intro- duce WIKICREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pro- noun resolution in combination with our WI- KICREM dataset. We compare a series of models on a collection of diverse and challeng- ing coreference resolution problems, where we match or outperform previous state-of-the- art approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WINOBIAS, and WINOGENDER. We release our model to be used off-the-shelf for solving pronoun disam- biguation. 1 Introduction Pronoun resolution, also called coreference or ana- phora resolution, is a natural language processing (NLP) task, which aims to link the pronouns with their referents. This task is of crucial importance in various other NLP tasks, such as information extraction (Nakayama, 2019) and machine trans- lation (Guillou, 2012). Due to its importance, pro- noun resolution has seen a series of different ap- proaches, such as rule-based systems (Lee et al., 2013) and end-to-end-trained neural models (Lee et al., 2017; Liu et al., 2019). However, the re- cently released dataset GAP (Webster et al., 2018) shows that most of these solutions perform worse than na¨ıve baselines when the answer cannot be deduced from the syntax. Addressing this draw- back is difficult, partially due to the lack of large- scale challenging datasets needed to train the data- hungry neural models. As observed by Trinh and Le (2018), language models are a natural approach to pronoun resolu- tion, by selecting the replacement for a pronoun that forms the sentence with highest probability. Additionally, language models have the advantage of being pre-trained on a large collection of un- structured text and then fine-tuned on a specific task using much less training data. This proce- dure has obtained state-of-the-art results on a se- ries of natural language understanding tasks (De- vlin et al., 2018). In this work, we address the lack of large train- ing sets for pronoun disambiguation by introduc- ing a large dataset that can be easily extended. To generate this dataset, we find passages of text where a personal name appears at least twice and mask one of its non-first occurrences. To make the disambiguation task more challenging, we also ensure that at least one other distinct personal name is present in the text in a position before the masked occurrence. We instantiate our method on English Wikipedia and generate the Wikipedia …

Content truncated to 500 words