Evaluation of Off-the-shelf Speech Recognizers on Different Accents in a Dialogue Domain

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6001–6008 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6001 Evaluation of Off-the-shelf Speech Recognizers on Different Accents in a Dialogue Domain Divya Tadimeti1,2, Kallirroi Georgila2, David Traum2 1Electrical Engineering & Computer Science, University of California, Berkeley 2Institute for Creative Technologies, University of Southern California dtadimeti@berkeley.edu, {kgeorgila,traum}@ict.usc.edu Abstract We evaluate several publicly available off-the-shelf (commercial and research) automatic speech recognition (ASR) systems on dialogue agent-directed English speech from speakers with General American vs. non-American accents. Our results show that the performance of the ASR systems for non-American accents is considerably worse than for General American accents. Depending on the recognizer, the absolute difference in performance between General American accents and all non-American accents combined can vary approximately from 2% to 12%, with relative differences varying approx- imately between 16% and 49%. This drop in performance becomes even larger when we consider specific categories of non-American accents indicating a need for more diligent collection of and training on non-native English speaker data in order to narrow this performance gap. There are performance differences across ASR systems, and while the same general pattern holds, with more errors for non-American accents, there are some accents for which the best recognizer is different than in the overall case. We expect these results to be useful for dialogue system designers in developing more robust inclusive dialogue systems, and for ASR providers in taking into account performance requirements for different accents. Keywords: speech recognition evaluation, accents, dialogue systems 1. Introduction Automatic speech recognition (ASR) systems are being used for an increasing number of speech-to-text appli- cations. With this proliferation, it is increasingly im- portant for this technology to serve all subgroups of consumers. Recent work has shown that ASR systems have a much higher error rate on speakers of African American Vernacular English (AAVE) than on rural White Californians engaging in sociolinguistic inter- views (Koenecke et al., 2020). Recent evaluation of ASR systems on speech directed at computer agents (Georgila et al., 2020) shows that speech recogniz- ers have been getting better recently on agent-directed speech compared to previous years (Yao et al., 2010; Morbini et al., 2013), but leaves open the question of whether this performance is equivalent for different speakers or whether the pattern observed by (Koenecke et al., 2020) also holds for other kinds of accents and for agent-directed speech. To this end, we evaluate the performance of popular ASR platforms — Google, Microsoft, Apple, Amazon, IBM, and Kaldi — on English speech from populations with different accents. We begin with a high-level dis- tinction between General American accents and non- American accents, and then focus on more specific cat- egories of non-American accents including French, In- dian, British, and East Asian accents. We report on a re-analysis of a subset of the ASR outputs examined by Georgila et al. (2020), including ASR outputs using 2 additional configurations of the Google ASR platform that were …

Content truncated to 500 words