Semi-Supervised Discriminative Language Modeling with Out-of-Domain Text Data

Proceedings of NAACL-HLT 2013, pages 727–732, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Semi-Supervised Discriminative Language Modeling with Out-of-Domain Text Data Arda C¸ elebi1 and Murat Sarac¸lar2 1Department of Computer Engineering 2Department of Electrical and Electronics Engineering Bo˘gazic¸i University, Istanbul, Turkey {arda.celebi, murat.saraclar}@boun.edu.tr Abstract One way to improve the accuracy of auto- matic speech recognition (ASR) is to use dis- criminative language modeling (DLM), which enhances discrimination by learning where the ASR hypotheses deviate from the uttered sentences. However, DLM requires large amounts of ASR output to train. Instead, we can simulate the output of an ASR sys- tem, in which case the training becomes semi- supervised. The advantage of using simu- lated hypotheses is that we can generate as many hypotheses as we want provided that we have enough text material. In typical scenar- ios, transcribed in-domain data is limited but large amounts of out-of-domain (OOD) data is available. In this study, we investigate how semi-supervised training performs with OOD data. We find out that OOD data can yield im- provements comparable to in-domain data. 1 Introduction Discriminative language modeling (DLM) helps ASR systems to discriminate between acoustically similar word sequences in the process of choos- ing the most accurate transcription of an utterance. DLM characterizes and learns from ASR errors by comparing the reference transcription of the utter- ance and the candidate hypotheses generated by the ASR system. Although previous studies based on this supervised setting have been successful (Roark et al., 2007; Arısoy et al., 2009; Arısoy et al., 2012; Sak et al., 2012), they require large amounts of tran- scribed speech data and a well-trained in-domain ASR system, both of which are hard to obtain. To overcome this difficulty, instead of training with the real ASR output, we can use simulated output, in which case the training becomes semi-supervised. Semi-supervised training for discriminative lan- guage modeling has been shown to achieve as good word error rate (WER) reduction as the training done with real ASR output (Sagae et al., 2012; C¸ elebi et al., 2012). In this approach, first a confusion model (CM) is estimated from supervised data. This CM contains all seen confusions and their occurrence probabilities in hypotheses generated by an ASR system. Then, the CM is used to generate a num- ber of alternative-but-incorrect hypotheses, or simu- lated hypotheses, for a given sentence. Since the CM characterizes the errors that the ASR system makes, simulated hypotheses carry these characteristics. At the end, the DLM is trained on the reference sen- tences and their simulated hypotheses. Although be- ing able to simulate the output of the ASR system allows us to generate as much output as we need for the DLM training, there is not always enough text data that is in the same domain as the ASR sys- tem. Yet, it is easier to find large amounts of out-of- domain (OOD) text data. In this study, we extend the previous studies where in-domain text data was used for hypothesis simulation. …

Content truncated to 500 words