Generalizing and Hybridizing Count-based and Neural Language Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1163–1172, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Generalizing and Hybridizing Count-based and Neural Language Models Graham Neubig† and Chris Dyer‡ †Carnegie Mellon University, USA ‡Google DeepMind, United Kingdom Abstract Language models (LMs) are statistical mod- els that calculate probabilities over sequences of words or other discrete symbols. Currently two major paradigms for language model- ing exist: count-based n-gram models, which have advantages of scalability and test-time speed, and neural LMs, which often achieve superior modeling performance. We demon- strate how both varieties of models can be uni- fied in a single modeling framework that de- fines a set of probability distributions over the vocabulary of words, and then dynamically calculates mixture weights over these distri- butions. This formulation allows us to create novel hybrid models that combine the desir- able features of count-based and neural LMs, and experiments demonstrate the advantages of these approaches.1 1 Introduction Language models (LMs) are statistical models that, given a sentence wI 1 := w1, . . . , wI, calculate its probability P(wI 1). LMs are widely used in applica- tions such as machine translation and speech recog- nition, and because of their broad applicability they have also been widely studied in the literature. The most traditional and broadly used language model- ing paradigm is that of count-based LMs, usually smoothed n-grams (Witten and Bell, 1991; Chen 1Work was performed while GN was at the Nara Institute of Science and Technology and CD was at Carnegie Mellon Uni- versity. Code and data to reproduce experiments is available at http://github.com/neubig/modlm and Goodman, 1996). Recently, there has been a fo- cus on LMs based on neural networks (Nakamura et al., 1990; Bengio et al., 2006; Mikolov et al., 2010), which have shown impressive improvements in performance over count-based LMs. On the other hand, these neural LMs also come at the cost of in- creased computational complexity at both training and test time, and even the largest reported neural LMs (Chen et al., 2015; Williams et al., 2015) are trained on a fraction of the data of their count-based counterparts (Brants et al., 2007). In this paper we focus on a class of LMs, which we will call mixture of distributions LMs (MODLMs; §2). Specifically, we define MODLMs as all LMs that take the following form, calculat- ing the probabilities of the next word in a sentence wi given preceding context c according to a mix- ture of several component probability distributions Pk(wi|c): P(wi|c) = K � k=1 λk(c)Pk(wi|c). (1) Here, λk(c) is a function that defines the mixture weights, with the constraint that �K k=1 λk(c) = 1 for all c. This form is not new in itself, and widely used both in the calculation of smoothing coeffi- cients for n-gram LMs (Chen and Goodman, 1996), and interpolation of LMs of various varieties (Je- linek and Mercer, 1980). The main contribution of this paper is to demon- strate that …

Content truncated to 500 words