Error Detection in Automatic Speech Recognition

Error Detection in Automatic Speech Recognition Farshid Zavareh, Ingrid Zukerman, Su Nam Kim and Thomas Kleinbauer Faculty of Information Technology, Monash University Clayton, VICTORIA 3800, Australia sfhos2@student.monash.edu, {Ingrid.Zukerman,Su.Kim,Thomas.Kleinbauer}@monash.edu Abstract We offer a supervised machine learn- ing approach for recognizing erroneous words in the output of a speech recognizer. We have investigated several sets of fea- tures combined with two word configura- tions, and compared the performance of two classifiers: Decision Trees and Naïve Bayes. Evaluation was performed on a cor- pus of 400 spoken referring expressions, with Decision Trees yielding a high recog- nition accuracy. 1 Introduction One of the main stumbling blocks for spoken Nat- ural Language Understanding (NLU) systems is the lack of reliability of Automatic Speech Rec- ognizers (ASRs) (Pellegrini and Trancoso, 2010). Recent research prototypes of ASRs yield Word Error Rates (WERs) between 15.6% (Pellegrini and Trancoso, 2010) and 18.7% (Sainath et al., 2011) for broadcast news. However, the WER of the ASR we employed (Microsoft Speech SDK 6.1) is 34% when trained on an open vocabulary plus a small language model for our corpus. This WER is consistent with that obtained in the 2010 Spoken Dialogue Challenge (Black et al., 2011). In this paper, we offer a supervised machine learning approach to detect erroneous words in ASR output (this step will be followed by auto- matic error correction). Our approach was evalu- ated on a corpus of 400 spoken referring expres- sions, with the best-performing option yielding an average accuracy of 89% (Section 5). The rest of this paper is organized as follows. In the next section, we discuss related work. In Sec- tion 4, we describe our experimental design, fo- cusing on the features considered for our machine- learning approach. In Section 5, we discuss our re- sults, followed by concluding remarks. 2 Related Research Approaches for improving the performance of spoken NLU systems may be classified into pre- vention and recovery. Prevention avoids errors by constraining the vocabulary (Gorniak and Roy, 2005; Sugiura et al., 2009) and grammatical constructs (Brooks and Breazeal, 2006) understood by an ASR. ASRs that employ this approach can process expected utter- ances efficiently, and work well in restricted do- mains. However, these ASRs have trouble process- ing unexpected utterances. Recovery involves error detection followed by correction. During detection, an NLU system posits that a word in an utterance was incorrectly recognized. Three approaches to error recovery are described in (López-Cózar and Griol, 2010; Ringger and Allen, 1996; Zhou et al., 2006). López-Cózar and Griol (2010) consider statis- tical information, and lexical, syntactic, semantic and dialogue-related information to correct ASR errors (i.e., replace, insert or delete words in a textual ASR output), and syntactic approaches to modify tenses of verbs and grammatical numbers to better match grammatical expectations. Ringger and Allen (1996) use statistical infor- mation to construct a language model that quanti- fies the likelihood of word sequences, and a noisy channel model that predicts errors made by an ASR. They perform error detection and correction at the …

Content truncated to 500 words