Assessing the Quality of an Italian Crowdsourced Idiom Corpus:the Dodiom Experiment

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4205–4211 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4205 Assessing the Quality of an Italian Crowdsourced Idiom Corpus: the Dodiom Experiment Giuseppina Morza, Raffaele Manna, Johanna Monti UNIOR NLP Research Group - University of Naples “L’Orientale” Via Duomo 219, Naples, Italy {gmorza, rmanna, jmonti}@unior.it Abstract This paper focuses on the evaluation of linguistic data, concerning idioms examples collected and annotated through Dodiom, a GWAP environment, by Italian linguists. The paper provides an insight into the Dodiom project, the data collection through the contribution of the crowd, and, finally it specifically describes the annotation criteria used by the experts to estimate the quality of the collected data. The main scope of this paper is, indeed, the evaluation of the quality of the linguistic data obtained through crowdsourcing, namely to assess if the data provided by the players who joined the game are eligible and profitable for research and teaching purposes. This task concerns the development of a collection of idioms, namely a specific type of Multiword expressions which is usually hard to find in corpora and that contains words that may also be used in their literal meanings within a sentence. This is particularly important as these data may be used both for the training and the evaluation of NLP applications. Finally, results, as well as future work, are presented. Keywords: crowdsourcing, data quality, idioms 1. Introduction Natural Language Processing (NLP) has made signif- icant advances in recent years, due to the introduction of statistical machine learning techniques. Human an- notators, language experts, obviously play an essential role in the process of building language corpora, re- quired to train statistical learners. However, this un- avoidably results in high annotation costs and limited access to qualified annotators, which is a major hin- drance to NLP research. As shown by popular plat- forms like Wikipedia, Duolingo among others, crowd- sourcing presents a unique opportunity to obtain mas- sive amounts of data in a relatively short time. The research task described in this contribution is based on (Eryi˘git et al., 2021), who resorted to increasingly pop- ular methodologies for data collection, namely crowd- sourcing and gamification. The scholars developed a Game-with-a-purpose (GWAP) named Dodiom to elicit idiom examples and ratings from non-expert la- belers reached via Facebook, Linkedin and word-of- mouth. For instance, given a MWE, Dodiom asks the non-expert user to create a sentence in which the MWE is used idiomatically or not and to provide a label re- lated to one of the two uses (literal or idiomatic). Our aim was to assess by means of a subsequent an- notation task of the collected textual samples if the data obtained from non-experts (the crowd) in a cost- effective and time-saving way were valuable resources that could effectively be used as training and testing data for language understanding, in general, and idiom identification systems, as well as language learning ma- terial, or samples for …

Content truncated to 500 words