The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3952–3961 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3952 The KNOWREF Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution Ali Emami*1, Paul Trichelair*1, Adam Trischler2, Kaheer Suleman2, Hannes Schulz2, and Jackie Chi Kit Cheung1 1School of Computer Science, Mila/McGill University 2Microsoft Research Montreal {ali.emami, paul.trichelair}@mail.mcgill.ca {adam.trischler, kasulema, hannes.schulz}@microsoft.com jcheung@cs.mcgill.ca Abstract We introduce a new benchmark for corefer- ence resolution and NLI, KNOWREF, that tar- gets common-sense understanding and world knowledge. Previous coreference resolution tasks can largely be solved by exploiting the number and gender of the antecedents, or have been handcrafted and do not reflect the diver- sity of naturally occurring text. We present a corpus of over 8,000 annotated text pas- sages with ambiguous pronominal anaphora. These instances are both challenging and re- alistic. We show that various coreference systems, whether rule-based, feature-rich, or neural, perform significantly worse on the task than humans, who display high inter- annotator agreement. To explain this perfor- mance gap, we show empirically that state-of- the art models often fail to capture context, in- stead relying on the gender or number of can- didate antecedents to make a decision. We then use problem-specific insights to propose a data-augmentation trick called antecedent switching to alleviate this tendency in mod- els. Finally, we show that antecedent switch- ing yields promising results on other tasks as well: we use it to achieve state-of-the-art re- sults on the GAP coreference task. 1 Introduction Coreference resolution is one of the best known tasks in Natural Language Processing (NLP). De- spite a large body of work in the area over the last few decades (Morton, 2000; Bean and Riloff, 2004; McCallum and Wellner, 2005; Rahman and Ng, 2009), the task remains challenging. Many resolution decisions require extensive world knowl- edge and understanding common points of refer- ence (Pradhan et al., 2011). In the case of pronomi- nal anaphora resolution, these forms of “common sense” become much more important when cues *equal contribution like gender and number do not by themselves indi- cate the correct resolution (Trichelair et al., 2018). To date, most existing methods for coreference resolution (Raghunathan et al., 2010; Lee et al., 2011; Durrett et al., 2013; Lee et al., 2017, 2018) have been evaluated on a few popular datasets, in- cluding the CoNLL 2011 and 2012 shared corefer- ence resolution tasks (Pradhan et al., 2011, 2012). These datasets were proposed as the first compre- hensively tagged and large-scale corpora for coref- erence resolution, to spur progress in state-of-the- art techniques. According to Durrett and Klein (2013), this progress would contribute in the “up- hill battle” of modelling not just syntax and dis- course, but also semantic compatibility based on world knowledge and context. Despite improvements in benchmark dataset per- formance, the question of what exactly current sys- tems learn or exploit remains open, particularly with recent neural coreference resolution models. Lee et al. (2017) …

Content truncated to 500 words