A Simple Language Model based on PMI Matrix Approximations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1860–1865 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Simple Language Model based on PMI Matrix Approximations Oren Melamud IBM Research Yorktown Heights, NY, USA oren.melamud@ibm.com Ido Dagan Computer Science Dept. Bar-Ilan University, Israel dagan@cs.biu.ac.il Jacob Goldberger Faculty of Engineering Bar-Ilan University, Israel jacob.goldberger@biu.ac.il Abstract In this study, we introduce a new approach for learning language models by training them to estimate word-context pointwise mutual information (PMI), and then de- riving the desired conditional probabili- ties from PMI at test time. Specifically, we show that with minor modifications to word2vec’s algorithm, we get principled language models that are closely related to the well-established Noise Contrastive Es- timation (NCE) based language models. A compelling aspect of our approach is that our models are trained with the same sim- ple negative sampling objective function that is commonly used in word2vec to learn word embeddings. 1 Introduction Language models (LMs) learn to estimate the prob- ability of a word given a context of preceding words. Recurrent Neural Network (RNN) language models recently outperformed traditional n-gram LMs across a range of tasks (Jozefowicz et al., 2016). However, an important practical issue asso- ciated with such neural-network LMs is the high computational cost incurred. The key factor that limits the scalability of traditional neural LMs is the computation of the normalization term in the softmax output layer, whose cost is linearly propor- tional to the size of the word vocabulary. Several methods have been proposed to cope with this scaling issue by replacing the softmax with a more computationally efficient component at train time.1 These include importance sam- 1An alternative recent approach for coping with large word vocabularies is to represent words as compositions of sub- word units, such as individual characters. This approach has notable merits (Jozefowicz et al., 2016; Sennrich et al., 2016), but is out of the scope of this paper. pling (Bengio and et al, 2003), hierarchical softmax (Minh and Hinton, 2008), BlackOut (Ji et al., 2016) and Noise Contrastive Estimation (NCE) (Gutmann and Hyvarinen, 2012). NCE has been applied to train neural LMs with large vocabularies (Mnih and Teh, 2012) and more recently was also suc- cessfully used to train LSTM-RNN LMs (Vaswani et al., 2013; Chen et al., 2015; Zoph et al., 2016). NCE-based language models achieved near state- of-the-art performance on language modeling tasks (Jozefowicz et al., 2016; Chen et al., 2016), and as we later show, are closely related to the method presented in this paper. Continuous word embeddings were initially in- troduced as a ‘by-product’ of learning neural lan- guage models (Bengio and et al, 2003). However, they were later adopted in many other NLP tasks, and the most popular recent word embedding learn- ing models are no longer proper language models. In particular, the skip-gram with negative sampling (NEG) embedding algorithm (Mikolov et al., 2013) as implemented in the word2vec toolkit, has be- come one of the most popular …

Content truncated to 500 words