Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3041–3052, Osaka, Japan, December 11-17 2016. Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax Bruno Guillaume Inria Nancy Grand-Est/LORIA bruno.guillaume@inria.fr Kar¨en Fort Universit´e Paris-Sorbonne / STIH karen.fort@paris-sorbonne.fr Nicolas Lefebvre Inria Nancy Grand-Est/LORIA nicolas.lefebvre@inria.fr Abstract This article presents the results we obtained on a complex annotation task (that of dependency syntax) using a specifically designed Game with a Purpose, ZombiLingo.1 We show that with suitable mechanisms (decomposition of the task, training of the players and regular control of the annotation quality during the game), it is possible to obtain annotations whose quality is significantly higher than that obtainable with a parser, provided that enough players participate. The source code of the game and the resulting annotated corpora (for French) are freely available. 1 Introduction For better or worse,2 the overwhelming domination of machine-learning systems in natural language processing (NLP) over the past 25 years and the increasing use of evaluation campaigns and shared tasks have put human-annotated corpora at the heart of the field. Manual annotation is now the place where linguistics hides. The availability of manually annotated corpora of high quality (or, at least, reliability) is therefore key to the development of the field in any given language. However, the creation of such resources is notoriously costly, especially when complex annotations, e.g. for dependency syntax, are at issue. For example, the cost of the Prague Dependency Treebank was estimated at $600,000 in (B¨ohmov´a et al., 2001). Over the years, many solutions have been investigated in the attempt to lower manual annotation costs. One obvious avenue is to use an appropriate annotation tool, as shown for example in (Dandapat et al., 2009). As a complementary aid, NLP systems can be used to reduce the annotation burden – either beforehand, for example with tag dictionaries (Carmen et al., 2010) and more generally with pre- annotation (Skjærholt, 2013), or more iteratively during the annotation process, for instance through active learning (Baldridge and Osborne, 2004). These solutions have proved efficient and have indeed helped reduce the annotation cost, but the creation of a large annotated corpus in the traditional manner remains very expensive. Another way to address the issue is simply to limit the amount paid to the human annotators. This is the case with microworking crowdsourcing, especially through the use of platforms like Amazon Mechanical Turk, via which the workers are (micro)paid to perform simplified tasks (Snow et al., 2008). Apart from the ethical issues raised by these platforms (detailed in (Fort et al., 2011)), microworking platforms do not support true training of the workers: tests can be set up to select them, but they have to perform hidden work to train themselves (as shown in (Gupta et al., 2014)). Consequently, tasks must be simplified to be manageable by workers: for example, a task related to recognition of textual entailment across several sentences (which might actually reflect entailment, neutrality, or contradiction) 1See: http://zombilingo.org/. This work is licenced under …

Content truncated to 500 words