The Universal Anaphora Scorer

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4873–4883 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4873 The Universal Anaphora Scorer Juntao Yu1, Sopan Khosla2,3,*, Nafise Sadat Moosavi4, Silviu Paun5, Sameer Pradhan6,7 and Massimo Poesio5 1Univ. of Essex, UK; 2Carnegie Mellon Univ., USA; 3AWS AI, Amazon, USA; 4Univ. of Sheffield, UK; 5Queen Mary Univ., UK; 6LDC, Univ. of Pennsylvania, USA; 7cemantix.org j.yu@essex.ac.uk; sopankh@amazon.com; m.poesio@qmul.ac.uk Abstract The aim of the Universal Anaphora initiative is to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. In this paper, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate the extended range of anaphoric interpretation included in the current Universal Anaphora proposal. The UA scorer supports the evaluation of identity anaphora resolution and of bridging reference resolution, for which scorers already existed but not integrated in a single package. It also supports the evaluation of split antecedent anaphora and discourse deixis, for which no tools existed. The proposed approach to the evaluation of split antecedent anaphora is entirely novel; the proposed approach to the evaluation of discourse deixis leverages the encoding of discourse deixis proposed in Universal Anaphora to enable the use for discourse deixis of the same metrics already used for identity anaphora. The scorer was tested in the recent CODI-CRAC 2021 Shared Task on Anaphora Resolution in Dialogues. Keywords: Anaphora Resolution, Evaluation, Univeral Anaphora 1. Introduction The performance of models for single-antecedent anaphora resolution on the aspects of anaphoric in- terpretation annotated in the reference ONTONOTES dataset (Pradhan et al., 2012) has greatly improved in recent years (Wiseman et al., 2015; Lee et al., 2017; Lee et al., 2018; Kantor and Globerson, 2019; Joshi et al., 2020). So the attention of the community has started to turn to more complex cases of anaphora not represented in ONTONOTES. Well-known examples of this trend are work on the cases of anaphora whose interpretation requires some form of commonsense knowledge tested by bench- marks for the Winograd Schema Challenge (Rahman and Ng, 2012; Liu et al., 2017; Sakaguchi et al., 2020), or the pronominal anaphors that cannot be resolved purely using gender, for which benchmarks such as GAP have been developed (Webster et al., 2018). An- other fruitful line of research has been devoted to cre- ating datasets covering genres other than news, such as conversation (Muzerelle et al., 2014; Uryupina et al., 2020; Khosla et al., 2021), fiction (Bamman et al., 2020) or scientific articles (Cohen et al., 2017). Further research …

Content truncated to 500 words