ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution

Findings of the Association for Computational Linguistics: EACL 2023, pages 312–330 May 2-6, 2023 ©2023 Association for Computational Linguistics ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution Ankita Gupta♠ Marzena Karpinska♠ Wenlong Zhao♠ Kalpesh Krishna♠ Jack Merullo♢ Luke Yeh♡ Mohit Iyyer♠ Brendan O’Connor♠ ♠University of Massachusetts Amherst, ♢Brown University, ♡Google {ankitagupta,mkarpinska,wenlongzhao,kalpesh,miyyer,brenocon}@cs.umass.edu john_merullo@brown.edu, lukeyeh@google.com Abstract Large-scale, high-quality corpora are critical for advancing research in coreference reso- lution. However, existing datasets vary in their definition of coreferences and have been collected via complex and lengthy guidelines that are curated for linguistic experts. These concerns have sparked a growing interest among researchers to curate a unified set of guidelines suitable for annotators with vari- ous backgrounds. In this work, we develop a crowdsourcing-friendly coreference annota- tion methodology, ezCoref, consisting of an annotation tool and an interactive tutorial. We use ezCoref to re-annotate 240 passages from seven existing English coreference datasets (spanning fiction, news, and multiple other do- mains) while teaching annotators only cases that are treated similarly across these datasets.1 Surprisingly, we find that reasonable quality an- notations were already achievable (>90% agree- ment between crowd and experts) even with- out extensive training. On carefully analyzing the remaining disagreements, we identify the presence of linguistic cases that our annotators unanimously agree upon but lack unified treat- ments (e.g., generic pronouns, appositives) in existing datasets. We propose the research com- munity should revisit these phenomena when curating future unified annotation guidelines. 1 Introduction Coreference resolution is the task of identifying and clustering together all textual expressions (men- tions) that refer to the same discourse entity in a given document. Impressive progress has been made in developing coreference systems (Lee et al., 2017; Moosavi and Strube, 2018; Joshi et al., 2020), enabled by datasets annotated by experts (Hovy et al., 2006; Bamman et al., 2020; Uryupina et al., 2019) and crowdsourcing (Chamberlain et al., 2016). However, these datasets vary widely in 1Our platform’s code and collected data is available at https://github.com/gnkitaa/ezCoref OntoNotes: Maybe we need a [CIA] version of the Miranda warning: You have the right to conceal your coup intentions, because we may rat on you. ARRAU: Maybe [we]e1 need [a [CIA] version of [the Miranda warning]]: [You]e4 have [the right to conceal [[your]e5 [coup] intentions]], because [we]e6 may rat on [you]e7. Crowd (this work): Maybe [we]e1 need [a [CIA] version of [the [Miranda] warning]]: [You]e3 have [the right] to conceal [[your]e3 coup intentions], because [we]e1may rat on [you]e3. Figure 1: We visualize a common sentence from news domain annotated by two expert-curated datasets, OntoNotes (Hovy et al., 2006) and ARRAU (Uryupina et al., 2019), along with the crowd annotations collected via our ezCoref platform. OntoNotes does not mark generic pronouns. ARRAU does not consider them as coreferent and annotates them using a special re- lation “undef-reference” (markables with vague inter- pretations). On the contrary, our crowdworkers assign all mentions of the generic pronoun “you” to the same coreference chain. The situation is also similar for the generic “we.” their definitions of coreference (expressed …

Content truncated to 500 words