Rigor Mortis: Annotating MWEs with a Gamified Platform

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4395–4401 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4395 Rigor Mortis: Annotating MWEs with a Gamified Platform Kar¨en Fort∗†, Bruno Guillaume†, Yann-Alan Pilatte∗, Mathieu Constant‡, Nicolas Lef`ebvre† ∗Sorbonne-Universit´e / STIH, †Universit´e de Lorraine, CNRS, Inria, LORIA, ‡Universit´e de Lorraine, CNRS, ATILF 28, rue Serpente 75006 Paris, France, 54000 Nancy, France karen.fort@sorbonne-universite.fr, bruno.guillaume@inria.fr, yann-alan.pilatte@etu.sorbonne-universite.fr, mathieu.constant@univ-lorraine.fr, nico.nicolef@gmail.com Abstract We present here Rigor Mortis, a gamified crowdsourcing platform designed to evaluate the intuition of the speakers, then train them to annotate multi-word expressions (MWEs) in French corpora. We previously showed (Fort et al., 2018) that the speakers’ intuition is reasonably good (65% in recall on non-fixed MWE). After a training phase using some of the tests developed in the PARSEME-FR project, we obtain 0.685 in F-measure at an experimentally determined 25% threshold (number of players who annotated the same segment). Keywords: MWE, crowdsourcing, games with a purpose 1. Motivations Multiword expressions (MWEs) can roughly be defined as combinations of several words that encompass some com- position irregularity at one or several linguistic levels, in- cluding morphology, syntax and semantics. For instance, the meaning of the expression cut the mustard (succeed) cannot be derived from the meaning of its parts. Multiword expressions include multiple linguistic phenomena, as men- tioned in (Sag et al., 2001), for example idioms (e.g. add fuel to the fire), phrasal verbs (e.g. give up), complex func- tion words (e.g. as soon as), light-verb constructions (e.g. take a bath), adverbial and nominal open compounds (e.g. by the way, dry run). Handling MWEs is a key challenge for natural lan- guage processing (Sag et al., 2001), on which researchers have long been working. Recently, significant advances have been made thanks to the availability of new MWE- annotated data that are used to learn state-of-the-art iden- tification models (Schneider et al., 2016; Ramisch et al., 2018). The construction of such annotated corpora is nonetheless costly. Indeed, they are mainly annotated by experts or linguistics-aware people long-trained by experts in order to guarantee the annotation quality. Indeed, MWEs are known to be hard to identify due to the fuzzy delimitation be- tween compositional and non-compositional combinations of words. This difficulty is demonstrated in the modest av- erage inter-annotator agreement (0.65 in F-score) for com- prehensive MWEs annotation in English (Schneider et al., 2014). In this paper, we propose a gamified platform for annotating MWEs. Experiments were carried out for French. The aim of this paper is to assess to what extent one can rely on corpora annotated in MWEs by the participants with respect to experts. 2. Related Work The creation of linguistic resources for natural language processing using games with a purpose (GWAPs) has been tested for a while now. The first to be developed were JeuxDeMots, a game allowing the creation of a lexi- cal network for French, which is more than ten years old now (Lafourcade, 2007; Lafourcade et al., 2018), closely …

Content truncated to 500 words