Crowdsourcing a Multi-lingual Speech Corpus: Recording, Transcription and Annotation of the CrowdIS Corpora

Crowdsourcing a multilingual speech corpus: recording, transcription and annotation of the CROWDED CORPUS Andrew Caines1, Christian Bentz2, Calbert Graham1, Tim Polzehl3, Paula Buttery1 1The ALTA Institute, Dept. Theoretical & Applied Linguistics, University of Cambridge, U.K. 2Institute of Linguistics, Universität Tübingen, Germany 3Telekom Innovation Laboratories / Technische Universität Berlin, Germany apc38@cam.ac.uk, chris@christianbentz.de, crg29@cam.ac.uk, tim.polzehl@qu.tu-berlin.de, pjb48@cam.ac.uk Abstract We announce the release of the CROWDED CORPUS: a pair of speech corpora collected via crowdsourcing, containing a native speaker corpus of English (CROWDED_ENGLISH), and a corpus of German/English bilinguals (CROWDED_BILINGUAL). Release 1 of the CROWDED CORPUS contains 1000 recordings amounting to 33,400 tokens collected from 80 speakers and is freely available to other researchers. We recruited participants via the Crowdee application for Android. Recruits were prompted to respond to business-topic questions of the type found in language learning oral tests. We then used the CrowdFlower web application to pass these recordings to crowdworkers for transcription and annotation of errors and sentence boundaries. Finally, the sentences were tagged and parsed using standard natural language processing tools. We propose that crowdsourcing is a valid and economical method for corpus collection, and discuss the advantages and disadvantages of this approach. Keywords: crowdsourcing, speech corpus, annotation 1 Introduction We announce the release of two corpora collected via crowdsourcing: a native speaker corpus of English (CROWDED_ENGLISH), and a corpus of German/English bilinguals (CROWDED_BILINGUAL)1. We refer to these datasets as the CROWDED CORPUS, where Crowd refers to our data source, E = English, and D = Deutsch. Both corpora involve the speakers answering questions about se- lected business topics, and were motivated with the follow- ing two research questions in mind: a. With tasks and topics comparable to typical language learning oral exams, we can start to address the question, ‘what would a native speaker say in a similar situation?’ Hence we collected CROWDED_ENGLISH; b. With a corpus of the same speaker undertaking the same tasks in two languages, we can investigate the effects of first language transfer in terms of phonetics, lexis and syntax. Hence we collected CROWDED_BILINGUAL. It is well-known that building speech corpora2 is a time- consuming and expensive process: one estimate puts the cost of transcription at AC1 per word, before the cost of any extra annotation (Ballier and Martin, 2013). Presumably the main expense in this figure is researcher time – skilled labourers with accompanying overheads. Extending the crowdsourced transcription work described in (Cooper et al., 2014; van Dalen et al., 2015) by crowdsourcing both 1A poster with a similar title was presented at Corpus Linguis- tics 2015: since then we have extended our evaluations of data quality, collected more recordings, and made the first data release public. 2We note that an occasional distinction is made between ‘speech corpora’ and ‘spoken corpora’ (Ballier and Martin, 2013) but use the terms interchangeably here to mean ‘a collection of spoken/speech data’. recordings and transcriptions, we present a method to col- lect spoken language corpora via crowdsourcing facilities, showing that by distributing the work among multiple on- line …

Content truncated to 500 words