Crowdsourcing and Aggregating Nested Markable Annotations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 797–807 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 797 Crowdsourcing and Aggregating Nested Markable Annotations Chris Madge Queen Mary University c.j.madge@qmul.ac.uk Juntao Yu Queen Mary University juntao.yu@qmul.ac.uk Jon Chamberlain University of Essex jchamb@essex.ac.uk Udo Kruschwitz University of Essex udo@essex.ac.uk Silviu Paun Queen Mary University s.paun@qmul.ac.uk Massimo Poesio Queen Mary University m.poesio@qmul.ac.uk Abstract One of the key steps in language resource creation is the identification of the text seg- ments to be annotated, or markables– in our case, the (potentially nested) noun phrases in coreference resolution (or mentions). In this paper, we present a method for identify- ing markables for coreference annotation that combines high-performance automatic mark- able detectors with checking with a Game- With-A-Purpose (GWAP) and aggregation us- ing a Bayesian annotation model. The method was evaluated both on news data and data from a variety of other genres and results in an im- provement on F1 of mention boundaries of over seven percentage points when compared with a state-of-the-art, domain-independent automatic mention detector, and almost three points over an in-domain mention detector. One of the key contributions of our proposal is its applicability to the case in which mark- ables are nested, as is the case with corefer- ence markables; but the GWAP and several of the proposed markable detectors are task- and language-independent and are thus applicable to a variety of other annotation scenarios. 1 Introduction Developing Natural Language Processing (NLP) systems still requires large amounts of annotated text to train models, or as a gold standard to test the effectiveness of such models. The approach followed to create the most widely used data (Mar- cus et al., 1993; Palmer et al., 2005; Pradhan et al., 2012) is to separate the task of identifying the text segments to be annotated–the markables–from the annotation task proper. In our specific case, the markables of interest are the mentions used in coreference resolution, to be labelled as belonging to a coreference chain or as singletons; typical ex- amples of mentions are pronouns, named entities, and other nominal phrases (Poesio et al., 2016). The annotation of mentions for coreference has similarities with the identification of the chunks for named entity resolution (NER), but mentions can and often are nested, as in the following ex- ample, from the Phrase Detectives corpus (Cham- berlain et al., 2016)), where a mention of entity i is nested inside a mention of entity j. [A wolf]i had been gorging on [an animal [he]i had killed]j The methods proposed in this paper are also appli- cable when markables are nested. Mention identification for annotation is typi- cally done semi-automatically, using first an auto- matic mention detector (or extractor) (Uryupina and Zanoli, 2016) and then checking its output by hand. Automatic mention detectors developed for coreference systems are generally used in the first step. Mention detection was recognized early on as a key step for overall coreference quality (Stoy- anov …

Content truncated to 500 words