Speak: A Toolkit Using Amazon Mechanical Turk to Collect and Validate Speech Audio Recordings

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7253–7258 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7253 Speak: A Toolkit Using Amazon Mechanical Turk to Collect and Validate Speech Audio Recordings Christopher Song1, David Harwath2, Tuka Alhanai3, James Glass4 1Johns Hopkins University Baltimore, MD 21218, USA 2University of Texas at Austin Austin, TX 78712, USA 3New York University Abu Dhabi Abu Dhabi, UAE 4Massachusetts Institute of Technology Cambridge, MA 02139, USA 1csong23@jhu.edu, 2harwath@cs.utexas.edu, 3tuka.alhanai@nyu.edu, 4glass@mit.edu Abstract We present Speak, a toolkit that allows researchers to crowdsource speech audio recordings using Amazon Mechanical Turk (MTurk). Speak allows MTurk workers to submit speech recordings in response to a task prompt and stimulus (e.g. image, text excerpt, audio file) defined by researchers, a functionality that is not natively offered by MTurk at the time of writing this paper. Importantly, the toolkit employs multiple measures to ensure that speech recordings collected are of adequate quality, in order to avoid accepting unusable data and prevent abuse/fraud. Speak has demonstrated utility, having collected over 600,000 recordings to date. The toolkit is open-source and available for download. Keywords: data collection, speech recognition, human-computer interaction, crowdsourcing 1. Introduction Speech audio recordings remain an important source of data used in academic research. They are used in a variety of academic disciplines, including speech recognition, human computer interaction (Clark et al., 2019), neuroscience (Anumanchipalli et al., 2019; Ak- bari et al., 2019), psychology (Pouw et al., 2020), and medicine (Anfinrud et al., 2020; Karan et al., 2020; Alhanai et al., 2017). Given the continued relevance of speech data to academia, attempts to scale its col- lection are not new; researchers have collected speech recordings in large-scale efforts since the early 1990’s (Hirschman et al., 1993), but the challenge of collect- ing this data in a way that is both time- and resource- efficient continues to be a challenge. Technological development has both helped and sus- tained the issue. It has contributed to the evolution of collection efforts: from manual collection across collaborating institutions (Hirschman et al., 1993), to telephone-based collection (Zue et al., 1997), and even- tually to web-based collection (e.g. through open- source data, crowdsourcing). Crowdsourcing has par- ticularly shown promise as an efficient way to collect these recordings, with tools having previously been built (Saylor, 2015; McGraw et al., 2010) to source recordings from mainstream crowdsourcing platforms such as Amazon Mechanical Turk (MTurk). However, as some web technologies continue to de- velop, others stagnate. The dependencies underpin- ning these previous crowdsourcing tools have lost sup- port to the point of deprecation, thereby rendering the tools difficult to use for data collection. Researchers have continued to adopt other previously-established methods for speech data collection, such as soliciting recordings manually, or extracting them from open- source data; such solutions indeed have their own ben- efits, but they come with their own unique drawbacks as well (e.g. significant additional secondary efforts re- quired, lack of task-specificity, etc.) Crowdsourcing as a …

Content truncated to 500 words