PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5963–5973 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5963 PCR4ALL: A Comprehensive Evaluation Benchmark for Pronoun Coreference Resolution in English Xinran Zhao1,3, Hongming Zhang2,3, Yangqiu Song3 1Stanford University 2Tencent AI Lab, Seattle 3HKUST xzhaoar@stanford.edu, hongmzhang@tencent.com, yqsong@cse.ust.hk Abstract Pronoun Coreference Resolution (PCR) is the task of resolving pronominal expressions to all mentions they refer to. The correct resolution of pronouns typically involves the complex inference over both linguistic knowledge and general world knowledge. Recently, with the help of pre-trained language representation models, the community has made significant progress on various PCR tasks. However, as most existing works focus on developing PCR models for specific datasets and measuring the accuracy or F1 alone, it is still unclear whether current PCR systems are reliable in real applications. Motivated by this, we propose PCR4ALL, a new benchmark and a toolbox that evaluates and analyzes the performance of PCR systems from different perspectives (i.e., knowledge source, domain, data size, frequency, relevance, and polarity). Experiments demonstrate notable performance differences when the models are examined from different angles. We hope that PCR4ALL can motivate the community to pay more attention to solving the overall PCR problem and understand the performance comprehensively. All data and codes are available at: https://github.com/HKUST-KnowComp/PCR4ALL. Keywords: PCR, Format Unification, Multi-perspective Evaluation 1. Introduction The question of how human beings resolve pronouns has long been of interest to both linguistic and natural language processing (NLP) communities. The situa- tion that a pronoun itself only has weak semantic mean- ing brings challenges to natural language understand- ing systems. To explore solutions for that question, pronoun coreference resolution (PCR) (Hobbs, 1978) was proposed1. The first and second personal pronouns are typically not considered as they often refer to the current speakers, which are normally out of the con- versation or document. Conventional PCR works (Ng, 2005; Zhang et al., 2019a; Zhang et al., 2019b) mostly focus on identifying coreference relations between pro- nouns and noun phrases rather than relations between pronouns. As a challenging yet vital natural language understanding task, PCR is to find the correct refer- ence for a given pronominal anaphor in the context and has shown to be crucial for a series of down- stream tasks, including machine translation (Mitkov et al., 1995), summarization (Steinberger et al., 2007), in- formation extraction (Edens et al., 2003), and dialog systems (Strube and M¨uller, 2003). The correct resolution of pronouns typically requires reasoning over both linguistic knowledge (e.g., “they” can only refer to plural objects) and commonsense knowledge (e.g., in sentence “The fish ate the worm, 1Previous studies (Ng, 2005; Zhang et al., 2019b) mainly focus on three kinds of pronouns: third personal pronoun (e.g., she, her, he, him, them, they, it), possessive pronoun (e.g., his, hers, its, their, theirs), and demonstrative pronoun (e.g., this, that, these, those). it was hungry”, “it” refers to “fish” because hungry things tend to eat rather than being eaten). To inves- …

Content truncated to 500 words