Refining Generative Language Models using Discriminative Learning

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 51–58, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Refining Generative Language Models using Discriminative Learning Ben Sandbank Blavatnik School of Computer Science Tel-Aviv University Tel-Aviv 69978, Israel sandban@post.tau.ac.il Abstract We propose a new approach to language mod- eling which utilizes discriminative learning methods. Our approach is an iterative one: starting with an initial language model, in each iteration we generate 'false' sentences from the current model, and then train a clas- sifier to discriminate between them and sen- tences from the training corpus. To the extent that this succeeds, the classifier is incorpo- rated into the model by lowering the probabil- ity of sentences classified as false, and the process is repeated. We demonstrate the effec- tiveness of this approach on a natural lan- guage corpus and show it provides an 11.4% improvement in perplexity over a modified kneser-ney smoothed trigram. 1 Introduction Language modeling is a fundamental task in natu- ral language processing and is routinely employed in a wide range of applications, such as speech recognition, machine translation, etc’. Tradition- ally, a language model is a probabilistic model which assigns a probability value to a sentence or a sequence of words. We refer to these as generative language models. A very popular example of a generative language model is the n-gram, which conditions the probability of the next word on the previous (n-1)-words. Although simple and widely-applicable, it has proven difficult to allow n-grams, and other forms of generative language models as well, to take ad- vantage of non-local and overlapping features.1 These sorts of features, however, pose no problem for standard discriminative learning methods, e.g. large-margin classifiers. For this reason, a new class of language model, the discriminative lan- guage model, has been proposed recently to aug- ment generative language models (Gao et al., 2005; Roark et al., 2007). Instead of providing probability values, discriminative language models directly classify sentences as either correct or in- correct, where the definition of correctness de- pends on the application (e.g. grammatical / ungrammatical, correct translation / incorrect trans- lation, etc'). Discriminative learning methods require negative samples. Given that the corpora used for training language models contain only real sentences, i.e. positive samples, obtaining these can be problematic. In most work on discriminative language modeling this was not a major issue as the work was concerned with specific applications, and these provided a natural definition of negative samples. For instance, (Roark et al., 2007) proposed a discriminative language model for a speech recognition task. Given an acoustic sequence, a baseline recognizer was used to generate a set of possible transcriptions. The correct transcription was taken as a positive sample, while the rest were taken as negative samples. More recently, however, Okanohara and Tsujii (2007) showed that a 1 Conditional maximum entropy models (Rosenfeld, 1996) provide somewhat of a counter-example, but there, too, many kinds of global and non-local features are difficult to use (Rosenfeld, 1997). 51 discriminative …

Content truncated to 500 words