Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts

Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts David Alfer1, Therese Lindstr¨om Tiedemann2, Elena Volodina1 1University of Gothenburg 2University of Helsinki Spr˚akbanken Department of Finnish, Finno-Ugrian Department of Swedish and Scandinavian Studies david.alfter@gu.se therese.lindstromtiedemann@helsinki.fi elena.volodina@gu.se Abstract In this study we investigate to which degree experts and non-experts agree on questions of dificulty in a crowdsourcing experiment. We ask non-experts (second language learners of Swedish) and two groups of experts (teachers of Swedish as a second/foreign language and CEFR experts) to rank multi-word expressions in a crowdsourcing experiment. We find that the resulting rankings by all the three tested groups correlate to a very high degree, which suggests that judgments produced in a comparative seting are not influenced by professional insights into Swedish as a second language. 1 Introduction Many of the challenges in automatically driven solutions for language learning boil down to the lack of data and resources based on which we can develop language learning materials or train models. Re- sources like the English Vocabulary Profile (Capel, 2010, 2012; Cambridge University Press, 2015) are a luxury that cost a lot of time and resources to create, and for most languages such resources do not exist. Crowdsourcing has been suggested as one of the potential methods to overcome these challenges. Recently, a European network enet-Collect1 (Lyding et al., 2018) has been initiated to stimulate synergies between language learning research and practice on the one hand, and crowdsourcing on the other. New initiatives have arisen as a result, e.g. using implicitly crowdsourced learner knowledge for language resource creation (Nicolas et al., 2020), crowdsourc- ing corpus cleaning (Kuhn et al., 2019), development of the Learning and Read- ing Assistant LARA (Habibi, 2019). How- ever, there are many questions that need to be investigated and answered with regards to methodological issues arising from us- 1htps://enetcollect.eurac.edu/ ing crowdsourcing as a method in/for Lan- guage Learning. In this article, we raise some method- ological questions about crowdsourcing in the context of second language (L2) learn- ing material creation. To go back to the example of the English Vocabulary Profile – could we generate something similar for other languages without involving lexicog- raphers and experts? For example, given a set of some unordered vocabulary items (e.g. phrases), how can we order them by dificulty and split them into groups appro- priate for teaching at diferent levels of lin- guistic proficiency? Could a crowd help us in this scenario? Who can be “the crowd” in that case? How many answers are enough? How many contributors are needed? Are the results reliable? Parts of this article have been described in Alfer et al. (2020). We focus on whether a crowd of non-expert crowdsourcers can be used to generate language learning materials and how the annotations by experts such as L2 Swedish teachers, assessors and re- searchers, i.e. people with formal training in teaching and assessing in Swedish, com- pare to the annotations by non-experts, by which we mean learners and speakers of L2 Swedish.2 On a more …

Content truncated to 500 words