Work Hard, Play Hard: Collecting Acceptability Annotations through a 3D Game

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1740–1750 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1740 Work Hard, Play Hard: Collecting Acceptability Annotations through a 3D Game Federico Bonetti 1,2, Elisa Leonardelli1, Daniela Trotta3, Raffaele Guarasci4, Sara Tonelli1 1 Fondazione Bruno Kessler, Italy, 2 Universit`a di Trento, Italy 3 Universit`a di Salerno, Italy, 4 ICAR–CNR, Naples, Italy {fbonetti, eleonardelli, satonelli}@fbk.eu, dtrotta@unisa.it, raffaele.guarasci@icar.cnr.it Abstract Corpus-based studies on acceptability judgements have always stimulated the interest of researchers, both in theoretical and computational fields. Some approaches focused on spontaneous judgements collected through different types of tasks, others on data annotated through crowd-sourcing platforms, still others relied on expert annotated data available from the literature. The release of CoLA corpus, a large-scale corpus of sentences extracted from linguistic handbooks as examples of acceptable/non acceptable phenomena in English, has revived interest in the reliability of judgements of linguistic experts vs. non-experts. Several issues are still open. In this work, we contribute to this debate by presenting a 3D video game that was used to collect acceptability judgments on Italian sentences. We analyse the resulting annotations in terms of agreement among players and by comparing them with experts’ acceptability judgments. We also discuss different game settings to assess their impact on participants’ motivation and engagement. The final dataset containing 1,062 sentences, which were selected based on majority voting, is released for future research and comparisons. Keywords: acceptability, gamification, annotation 1. Introduction In recent years, studies on automatic assessment of ac- ceptability have become very popular thanks to the re- lease of the CoLA corpus (Warstadt et al., 2019), the first large-scale corpus of English acceptability, con- taining more than 10k sentences taken from linguis- tic literature, now included in the widely used GLUE benchmark (Wang et al., 2018). The corpus, whose Italian counterpart has been recently released (Trotta et al., 2021), has been developed based on the assumption that experts, i.e. linguists publishing handbooks and educational material on acceptability, have the required knowledge to define rules and examples to explain what is deemed acceptable or not in a language. In other terms, the example sentences reported in linguistic lit- erature can be used to build a gold standard and train systems that perform acceptability judgments. An- other strand of research related to acceptability, how- ever, claims that informal collection of acceptability judgments can guarantee high annotation quality, even when annotation is crowd-sourced through Amazon Mechanical Turk (AMT). Specifically, (Sprouse, 2011) shows that crowd-sourced annotations are almost indis- tinguishable from data annotated in a controlled envi- ronment by university students. Other works along the same line have focused on the role of crowd-sourcing using AMT (Lau et al., 2014; Sprouse et al., 2013). However, to our knowledge, no previous work has ex- plored the potential of gamification for acceptability annotation, which embeds the informal data collection criteria of AMT (i.e. non-expert annotation) while in- creasing annotators’ engagement. In this work we investigate the applicability of …

Content truncated to 500 words