RED-ACE: Robust Error Detection for ASR using Confidence Embeddings

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2800 - 2808 December 7-11, 2022 ©2022 Association for Computational Linguistics RED-ACE: Robust Error Detection for ASR using Confidence Embeddings Zorik Gekhman∗ Dina Zverinski∗ Jonathan Mallinson Genady Beryozkin Google Research {zorik,zverinski,jonmall,genady}@google.com Abstract ASR Error Detection (AED) models aim to post-process the output of Automatic Speech Recognition (ASR) systems, in order to de- tect transcription errors. Modern approaches usually use text-based input, comprised solely of the ASR transcription hypothesis, disregard- ing additional signals from the ASR model. In- stead, we utilize the ASR system’s word-level confidence scores for improving AED perfor- mance. Specifically, we add an ASR Con- fidence Embedding (ACE) layer to the AED model’s encoder, allowing us to jointly en- code the confidence scores and the transcribed text into a contextualized representation. Our experiments show the benefits of ASR confi- dence scores for AED, their complementary effect over the textual signal, as well as the effectiveness and robustness of ACE for com- bining these signals. To foster further research, we publish a novel AED dataset consisting of ASR outputs on the LibriSpeech corpus with annotated transcription errors.1 1 Introduction Automatic Speech Recognition (ASR) systems transcribe audio signals, consisting of speech, into text. While state-of-the-art ASR systems reached high transcription quality, training them requires large amounts of data and compute resources. For- tunately, many high performing systems are avail- able as off-the-shelf cloud services. However, a performance drop can be observed when applying them to specific domains or accents (Khandelwal et al., 2020; Mani et al., 2020), or when transcrib- ing noisy audio. Moreover, cloud services usually expose the ASR models as a black box, making it impossible to further fine-tune them. ASR Error Detection (AED) models are de- signed to post-process the ASR output, in order ∗ Equal contribution. 1Our code and data are available at https: //github.com/google-research/google-research/ tree/master/red-ace. Figure 1: Our AED pipeline. The confidence scores are quantized and jointly encoded with the transcription text into a contextualized representation. to detect transcription errors and avoid their propa- gation to downstream tasks (Errattahi et al., 2018). AED models are widely used in interactive systems, to engage the user to resolve the detected errors. For example, AED systems can be found in Google Docs Voice Typing, where low confidence words are underlined, making it easier for users to spot errors and take actions to correct them. Modern NLP models usually build upon the Transformer architecture (Vaswani et al., 2017). However, no Transformer-based AED models have been proposed yet. Recently, the Transformer has been applied to ASR error correction (Mani et al., 2020; Liao et al., 2020; Leng et al., 2021a,b), an- other ASR post-processing task. These models use only the transcription hypothesis text as input and discard other signals from the ASR model. How- ever, earlier work on AED (not Transformer-based) has shown the benefits of such signals (Allauzen, 2007; Pellegrini and Trancoso, 2009; Chen et al., 2013) and specifically the benefits of ASR word- level confidence …

Content truncated to 500 words