Palabras: Crowdsourcing Transcriptions of L2 Speech

Palabras: Crowdsourcing Transcriptions of L2 Speech Eric Sanders, Pepi Burgos, Catia Cucchiarini, Roeland van Hout CLS/CLST, Radboud University Nijmegen, the Netherlands {e.sanders, j.burgos, c.cucchiarini, r.vanhout}@let.ru.nl Abstract We developed a web application for crowdsourcing transcriptions of Dutch words spoken by Spanish L2 learners. In this paper we discuss the design of the application and the influence of metadata and various forms of feedback. Useful data were obtained from 159 participants, with an average of over 20 transcriptions per item, which seems a satisfactory result for this type of research. Informing participants about how many items they still had to complete, and not how many they had already completed, turned to be an incentive to do more items. Assigning participants a score for their performance made it more attractive for them to carry out the transcription task, but this seemed to influence their performance. We discuss possible advantages and disadvantages in connection with the aim of the research and consider possible lessons for designing future experiments. Keywords: crowdsourcing, transcription, L2 speech 1. Introduction In their research, (Burgos et al., 2013; Burgos et al., 2014; Burgos et al., 2015) studied the pronunciation of Dutch by Spanish L2 learners. Judgements of pronunciation quality were obtained from experts (Burgos et al., 2013; Burgos et al., 2014). However, judgements by nonexpert Dutch native listeners are also relevant and informative, as they can reveal which features of the learners’ vowel realizations may lead to confusions in perception. To get large numbers of transcriptions, it was decided to use crowdsourcing for data collection (Burgos et al., 2015). The use of crowd- sourcing to obtain annotations or scorings of intelligibility or accentedness of non-native speech is not new (Evanini et al., 2010; Cooke et al., 2013; Wang et al., 2013). For our purpose, we built a web application that allows partic- ipants to listen to utterances and transcribe what they hear. In their crowdsourcing experiment (Cooke et al., 2013) had observed that limited feedback could lead to low task en- gagement. For this reason, we decided to build in a few feedback parameters, to see whether the presence or type of feedback would impact the number and nature of the tran- scriptions. We also asked for metadata such as gender, age and completed education to be able to study how these vari- ables affect crowdsourcing behaviour/participation. The results of the analyses of the data that were collected with the application are reported in (Burgos et al., 2015). The current paper describes the application, the feedback parameters, metadata and their influence on the results. 2. Application Before designing the application, we defined a number of criteria aimed at maximizing response: • The application had to be easy to use. • The task had to be ”fun” to do. • It had to be shared on Facebook to attract new partici- pants. • Transcribers had to participate voluntarily. • Participants should be able to return and continue from where they left. This led to the web application that we called Palabras, …

Content truncated to 500 words