A Neural Model for Aggregating Coreference Annotation in Crowdsourcing

Proceedings of the 28th International Conference on Computational Linguistics, pages 5760–5773 Barcelona, Spain (Online), December 8-13, 2020 5760 A Neural Model for Aggregating Coreference Annotation in Crowdsourcing Maolin Li1,2, Hiroya Takamura2,3, Sophia Ananiadou1,2,4 1National Centre for Text Mining, The University of Manchester, Manchester, United Kingdom 2 Artificial Intelligence Research and Technology, National Institute of Advanced Industrial Science and Technology, Tokyo, Japan 3 Precision and Intelligence Laboratory, Tokyo Institute of Technology, Tokyo, Japan 4 The Alan Turing Institute, London, United Kingdom {maolin.li, sophia.ananiadou}@manchester.ac.uk takamura.hiroya@aist.go.jp Abstract Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a high- performance coreference resolution system is time-consuming and expensive. Crowdsourcing makes it possible to obtain the required amounts of data rapidly and cost-effectively. However, crowd-sourced labels can be noisy. To ensure high-quality data, it is crucial to infer the correct labels by aggregating the noisy labels. In this paper, we split the aggregation into two subtasks, i.e, mention classification and coreference chain inference. Firstly, we predict the general class of each mention using an autoencoder, which incorporates contextual information about each mention, while at the same time taking into account the mention’s annotation complexity and annotators’ reliability at different levels. Secondly, to determine the coreference chain of each mention, we use weighted voting which takes into account the learned reliability in the first sub- task. Experimental results demonstrate the effectiveness of our method in predicting the correct labels. We also illustrate our model’s interpretability through a comprehensive analysis of exper- imental results. 1 Introduction Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. However, it is time-consuming and expensive to collect the large amounts of data from expert annotators that are required to train high-performance coreference resolution systems. A rapid and cost- effective alternative is to obtain labels through crowdsourcing (Snow et al., 2008). However, crowd- sourced labels are often noisy. In the example in Figure 1, the mention it actually refers to The Super Lamb Banana. However, different crowd annotators produced conflicting labels for this mention. We can also observe that the coreference annotation is more complex than classification and sequence labels. Annotators have to determine an appropriate referent mention for some mentions. Because the perfor- mance of supervised learning models is highly dependent on the quality of training data, the aggregation of these noisy labels, (i.e., the process of determining the label that is most likely to be correct) is im- portant to obtain a high-quality training corpus. Although label aggregation is a well-studied topic, most existing studies of natural language labelling tasks have only focused on aggregating classification or sequence labels. To the best of our knowledge, there is only one previous study (Paun et al., 2018) that has investigated how to aggregate crowd-sourced coreference labels. In this paper, we propose a 2-step framework in which the aggregation task is broken down into two …

Content truncated to 500 words