Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6462–6468 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6462 Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications Josh Meyer,† Lynn Rauchenstein,† Joshua D. Eisenberg,† Nicholas Howell ◦ † Artie, Inc. ◦ Higher School of Economics Los Angeles, CA, USA Moscow, Russian Federation {josh.meyer, lynn.rauchenstein, joshua.eisenberg}@artie.com, nlhowell@gmail.com Abstract We describe the creation of the Artie Bias Corpus, an English dataset of expert­validated <audio, transcript> pairs with demographic tags for {age, gender, accent}. We also release open software which may be used with the Artie Bias Corpus to detect demographic bias in Automatic Speech Recognition systems and can be extended to other speech technologies. The Artie Bias Corpus is a curated subset of the Mozilla Common Voice corpus, which we release under a Creative Commons CC­0 license – the most open and permissive license for data. This article contains information on the criteria used to select and annotate the Artie Bias Corpus in addition to experiments in which we detect and attempt to mitigate bias in end­to­end speech recognition models. We observe a significant accent bias in our baseline DeepSpeech model, with more accurate transcriptions of US English compared to Indian English. We do not, however, find evidence for a significant gender bias. We then show significant improvements on individual demographic groups from fine­tuning. Keywords: speech corpus, automatic speech recognition, demographic bias, bias detection 1. Introduction Speech technologies such as Automatic Speech Recogni­ tion (ASR) and Speaker Verification have become an every­ day part of life in many countries. However, as the technol­ ogy becomes more common we are discovering how fragile it can be. Environmental noise is a clear source of perfor­ mance degradation, but other (less visible) sources of degra­ dation stem from demographic factors such as age, gen­ der, and accent (Hashimoto et al., 2018; Garnerin et al., 2019; Tatman, 2017; Tatman and Kasten, 2017). A speech technology exhibits demographic bias when performance is worse for one demographic group relative to another. Demographic sources of bias in speech technology are usu­ ally the result of imbalanced training datasets. Most re­ search and development in Automatic Speech Recognition for English has been performed on datasets comprised of a majority of white American English speakers. Even though this is a known problem, to the best of our knowledge there exist no free and open resources to diagnose (in an attempt to mitigate) this bias. The current lack of resources is the motivation behind the Artie Bias Corpus1 and associated tools2. 2. Prior work 2.1. Bias detection Previous work in the detection of demographic bias in Speech Recognition models can be found in Tatman (2017) and Tatman and Kasten (2017). In Tatman (2017) the author studies performance of the proprietary ASR systems used in Youtube automated captioning on a collection of “accent tag challenge” videos. The corpus used to identify bias was a crowd­sourced set of speakers self­identifying …

Content truncated to 500 words