Active Learning for Coreference Resolution using Discrete Annotation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8320–8331 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8320 Active Learning for Coreference Resolution using Discrete Annotation Belinda Z. Li†∗ Gabriel Stanovsky♠♦ Luke Zettlemoyer♠† ♠ University of Washington ♦Allen Institute for AI †Facebook belindali@fb.com {gabis,lsz}@cs.washington.edu Abstract We improve upon pairwise annotation for ac- tive learning in coreference resolution, by ask- ing annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. This simple modification, when combined with a novel mention clustering al- gorithm for selecting which examples to la- bel, is much more efficient in terms of the per- formance obtained per annotation budget. In experiments with existing benchmark corefer- ence datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour. Future work can use our annotation protocol to effectively develop coreference models for new domains. Our code is publicly available.1 1 Introduction Coreference resolution is the task of resolving anaphoric expressions to their antecedents (see Fig- ure 1). It is often required in downstream appli- cations such as question answering (Dasigi et al., 2019) or machine translation (Stanovsky et al., 2019). Exhaustively annotating coreference is an expensive process as it requires tracking corefer- ence chains across long passages of text. In news stories, for example, important entities may be ref- erenced many paragraphs after their introduction. Active learning is a technique which aims to reduce costs by annotating samples which will be most beneficial for the learning process, rather than fully labeling a large fixed training set. Active learning consists of two components: (1) a task- specific learning algorithm, and (2) an iterative sample selection algorithm, which examines the performance of the model trained at the previous iteration and selects samples to add to the annotated ∗*Work done while at the University of Washington. 1https://github.com/belindal/ discrete-active-learning-coref A volcano in Mexico, known to locals as Po-po , just started spewing molten rock. Are the two mentions coreferent? No What is the first appearance of the entity that the yellow- highlighted text refers to? A volcano in Mexico Figure 1: Discrete annotation. The annotator is shown the document, a span (yellow), and the span’s predicted antecedent (blue). In case the answer to the corefer- ence question is negative (i.e., the spans are not core- ferring), we present a follow-up question (“what is the first appearance of the entity?”), providing additional cost-effective signal. Our annotation interface can be seen in Figure 5 in the Appendix. training set. This method has proven successful for various tasks in low-resource domains (Garrette and Baldridge, 2013; Kholghi et al., 2015; Syed et al., 2016, 2017). Sachan et al. (2015) showed that active learn- ing can be employed for the coreference resolution task. They used gold data to simulate pairwise human-annotations, where two entity mentions are annotated as either coreferring or not (see first ques- tion in Figure 1). In this paper, we propose two improvements to active learning …

Content truncated to 500 words