Generating Task-Pertinent sorted Error Lists for Speech Recognition

Generating Task-Pertinent sorted Error Lists for Speech Recognition Olivier Galibert 1, Mohamed Ameur Ben Jannet 2, Juliette Kahn 1, Sophie Rosset 2 1LNE, F-78190 Trappes, France 2LIMSI, CNRS, Universit´e Paris-Saclay, F-91405 Orsay, France {first.last}@lne.fr, {first.last}@limsi.fr Abstract Automatic Speech recognition (ASR) is one of the most widely used components in spoken language processing applications. ASR errors are of varying importance with respect to the application, making error analysis keys to improving speech processing applications. Knowing the most serious errors for the applicative case is critical to build better systems. In the context of Automatic Speech Recognition (ASR) used as a first step towards Named Entity Recognition (NER) in speech, error seriousness is usually determined by their frequency, due to the use of the WER as metric to evaluate the ASR output, despite the emergence of more relevant measures in the literature. We propose to use a different evaluation metric form the literature in order to classify ASR errors according to their seriousness for NER. Our results show that the ASR errors importance is ranked differently depending on the used evaluation metric. A more detailed analysis shows that the estimation of the error impact given by the ATENE metric is more adapted to the NER task than the estimation based only on the most used frequency metric WER. Keywords: Automatic Speech Recognition, Metrics, Error Analysis 1. Introduction Automatic Speech recognition (ASR) is one of the most widely used components in spoken language processing ap- plications. Its outputs are a valuable source of features for downstream modules which try to reach the semantics of the message. Despite important progress, these systems still produce errors. Error analysis is one of the keys leading to better sys- tems. There exist a lot of work studying the errors of ASR systems (Rena Nemoto and Adda-Decker, 2008), using knowledge about them in order to improve ASR systems (Boh´aˇc et al., 2012; Dufour and Esteve, 2008) or trying to automatically detect them in ASR output (Ghannay et al., 2015). ASR errors have been mainly investigated in the frame- work of comparisons between automatic vs. human decod- ing of speech (Scharenborg, 2007; Lippmann, 1997). They pointed out that although today the best ASR speech mod- els are quite efficient, they have not yet reached the status of being able to perfectly take into account all the observed acoustic variation. Human listeners are still outperforming them by a factor of 5 to 6 (Vasilescu et al., 2012). The tax- onomy of errors pointed out that some words are frequently victims of ASR errors: in particular short, acoustically poor and frequent items lead to local ambiguity (Adda-Decker, 2006). Other work was done studying or classifying er- rors given the type of words which are involved. For ex- ample, (Goryainova et al., 2014) studied ASR errors given the Part of Speech of the associated word. Most of the studies done on ASR error analysis focus on the cause of errors more then on the possible impact that it can have on downstream …

Content truncated to 500 words