Discriminative Language Modeling with Conditional Random Fields and the Perceptron Algorithm

Discriminative Language Modeling with Conditional Random Fields and the Perceptron Algorithm Brian Roark Murat Saraclar AT&T Labs - Research {roark,murat}@research.att.com Michael Collins Mark Johnson MIT CSAIL Brown University mcollins@csail.mit.edu Mark Johnson@Brown.edu Abstract This paper describes discriminative language modeling for a large vocabulary speech recognition task. We con- trast two parameter estimation methods: the perceptron algorithm, and a method based on conditional random fields (CRFs). The models are encoded as determin- istic weighted finite state automata, and are applied by intersecting the automata with word-lattices that are the output from a baseline recognizer. The perceptron algo- rithm has the benefit of automatically selecting a rela- tively small feature set in just a couple of passes over the training data. However, using the feature set output from the perceptron algorithm (initialized with their weights), CRF training provides an additional 0.5% reduction in word error rate, for a total 1.8% absolute reduction from the baseline of 39.2%. 1 Introduction A crucial component of any speech recognizer is the lan- guage model (LM), which assigns scores or probabilities to candidate output strings in a speech recognizer. The language model is used in combination with an acous- tic model, to give an overall score to candidate word se- quences that ranks them in order of probability or plau- sibility. A dominant approach in speech recognition has been to use a “source-channel”, or “noisy-channel” model. In this approach, language modeling is effectively framed as density estimation: the language model’s task is to define a distribution over the source – i.e., the possible strings in the language. Markov (n-gram) models are of- ten used for this task, whose parameters are optimized to maximize the likelihood of a large amount of training text. Recognition performance is a direct measure of the effectiveness of a language model; an indirect measure which is frequently proposed within these approaches is the perplexity of the LM (i.e., the log probability it as- signs to some held-out data set). This paper explores alternative methods for language modeling, which complement the source-channel ap- proach through discriminatively trained models. The lan- guage models we describe do not attempt to estimate a generative model P(w) over strings. Instead, they are trained on acoustic sequences with their transcriptions, in an attempt to directly optimize error-rate. Our work builds on previous work on language modeling using the perceptron algorithm, described in Roark et al. (2004). In particular, we explore conditional random field meth- ods, as an alternative training method to the perceptron. We describe how these models can be trained over lat- tices that are the output from a baseline recognizer. We also give a number of experiments comparing the two ap- proaches. The perceptron method gave a 1.3% absolute improvement in recognition error on the Switchboard do- main; the CRF methods we describe give a further gain, the final absolute improvement being 1.8%. A central issue we focus on concerns feature selection. The number of distinct n-grams in our training data is close to 45 million, and …

Content truncated to 500 words