Robustness of end-to-end Automatic Speech Recognition Models – A Case Study using Mozilla DeepSpeech

Robustness of end-to-end Automatic Speech Recognition Models – A Case Study using Mozilla DeepSpeech Aashish Agarwal and Torsten Zesch Language Technology Lab University of Duisburg-Essen Duisburg, Germany Abstract When evaluating the performance of auto- matic speech recognition models, usually word error rate within a certain dataset is used. Special care must be taken in understanding the dataset in order to report realistic perfor- mance numbers. We argue that many perfor- mance numbers reported probably underesti- mate the expected error rate. We conduct ex- periments controlling for selection bias, gen- der as well as overlap (between training and test data) in content, voices, and recording conditions. We find that content overlap has the biggest impact, but other factors like gen- der also play a role. 1 Introduction Automatic Speech Recognition (ASR) has made striking progress in recent years with the deploy- ment of increasingly large deep neural networks (Zhang et al., 2017; Sperber et al., 2018; Chang et al., 2019; Zhang et al., 2020). Now when you see a shiny new model with an error rate reported to be below 10%, are you likely to get the same er- ror rate on your data? Many reported results prob- ably underestimate the word error rate (WER) to be expected when a model is applied outside of its exact training conditions (Likhomanenko et al., 2020) For example, in many datasets, there is a large imbalance between male and female voices (usu- ally not enough female data). When evaluating only within such a dataset and not controlling for gender, the model can optimize overall WER by performing worse for females (Tatman, 2017). If the model is eventually applied in a setting where males and females are equally likely to use the sys- tem, WER will be much higher. Other issues that might lead to underestimating error rate are overlaps between the train and test sets regarding content, voices or recording condi- tions. Another issue to be considered is selection bias when the training process can select samples for training and testing. A really robust model should generalize be- yond these factors, but we find that current mod- els trained on the available datasets do not. We argue that this is partly due to the focus on report- ing improvements in a within-dataset setting. It just sounds better to report a 4.3% WER on the standard dataset instead of a more realistic num- ber (which we show can be several times higher). However, as most real-world applications are un- likely to directly reflect the properties of a specific dataset, most users would be better off with more robust models and a realistic estimate. Most of the end-to-end speech recognition sys- tems for English use the Librispeech (Panayotov et al., 2015) corpus, which has pre-defined data splits trying to avoid the issues discussed above.1 For German data, standard splits are not fully es- tablished leading to large differences in WER be- tween datasets, e.g. Agarwal and Zesch (2019) re- port WER in the range between 15 …

Content truncated to 500 words