Language-specific Effects on Automatic Speech Recognition Errors for World Englishes

Proceedings of the 29th International Conference on Computational Linguistics, pages 7177–7186 October 12–17, 2022. 7177 Language-specific Effects on Automatic Speech Recognition Errors for World Englishes June Choe, Yiran Chen, May Pik Yu Chan, Aini Li, Xin Gao, Nicole Holliday Department of Linguistics, University of Pennsylvania, USA {yjchoe, cheny39, pikyu, liaini, kauhsin, nholl}@sas.upenn.edu Abstract Despite recent advancements in automated speech recognition (ASR) technologies, reports of unequal performance across speakers of dif- ferent demographic groups abound. At the same time, the focus on performance metrics such as the Word Error Rate (WER) in prior studies limit the specificity and scope of recom- mendations that can be offered for system engi- neering to overcome these challenges. The cur- rent study bridges this gap by investigating the performance of Otter’s automatic captioning system on native and non-native English speak- ers of different language background through a linguistic analysis of segment-level errors. By examining language-specific error profiles for vowels and consonants motivated by linguis- tic theory, we find that certain categories of errors can be predicted from the phonological structure of a speaker’s native language. 1 Introduction A central concern in the ethics of building natu- ral language processing (NLP) tools is ensuring equity in service and representation through a com- mitment to linguistic justice (e.g., Blodgett et al., 2020; Hovy and Prabhumoye, 2021). This issue is especially pertinent to automatic speech recog- nition (ASR) systems used to transcribe natural spoken language into text (Markl and McNulty, 2022). As ASR systems are increasingly being adopted into many aspects of social life (e.g., vir- tual assistants and automatic captioning), various concerns about the equity of ASR systems have been raised. Ideally, these systems should serve speakers of different demographic backgrounds equally well; however, the existing technologies are not entirely satisfactory. ASR performance has been found to vary by users’ dialect (Wheat- ley and Picone, 1991; Meyer et al., 2020), gender (Adda-Decker and Lamel, 2005; Tatman and Kas- ten, 2017; Tatman, 2017; Sawalha and Abushariah, 2013; Boito et al., 2022) and ethnicity (Koenecke et al., 2020; Martin and Tang, 2020). For instance, ASR systems designed for American English typ- ically perform worse for non-white speakers than for white speakers (Tatman and Kasten, 2017; Koe- necke et al., 2020). Such inequalities result in cer- tain groups of speakers being better represented than others, and may even further exacerbate exist- ing inequalities in society. At the same time, less is known about how the performance of ASR systems can vary for second- language (L2) English speakers, a particularly vul- nerable population of English speakers with di- verse backgrounds. A recent work by Chan et al. (2022) examined how Otter, a popular automatic transcription system, performs on L1 (native) and L2 (second-language) speakers of 24 English vari- eties. Not only do the English varieties supported by Otter have lower Word Error Rates (WER) com- pared to the unsupported varieties, gaps in perfor- mance is also driven by an independent effect of the structure of a speaker’s first language – Otter …

Content truncated to 500 words