Self-Normalization Properties of Language Modeling

Proceedings of the 27th International Conference on Computational Linguistics, pages 764–773 Santa Fe, New Mexico, USA, August 20-26, 2018. 764 Self-Normalization Properties of Language Modeling Jacob Goldberger Faculty of Engineering Bar-Ilan University, Israel jacob.goldberger@biu.ac.il Oren Melamud IBM Research Yorktown Heights, NY, USA oren.melamud@ibm.com Abstract Self-normalizing discriminative models approximate the normalized probability of a class with- out having to compute the partition function. In the context of language modeling, this property is particularly appealing as it may significantly reduce run-times due to large word vocabularies. In this study, we provide a comprehensive investigation of language modeling self-normalization. First, we theoretically analyze the inherent self-normalization properties of Noise Contrastive Estimation (NCE) language models. Then, we compare them empirically to softmax-based ap- proaches, which are self-normalized using explicit regularization, and suggest a hybrid model with compelling properties. Finally, we uncover a surprising negative correlation between self- normalization and perplexity across the board, as well as some regularity in the observed errors, which may potentially be used for improving self-normalization algorithms in the future. 1 Introduction The ability of statistical language models (LMs) to estimate the probability of a word given a context of preceding words, plays an important role in many NLP tasks, such as speech recognition and machine translation. Recurrent Neural Network (RNN) language models have recently become the preferred method of choice, having outperformed traditional n-gram LMs across a range of tasks (Jozefowicz et al., 2016). Unfortunately however, they suffer from scalability issues incurred by the computation of the softmax normalization term, which is required to guarantee proper probability predictions. The cost of this computation is linearly proportional to the size of the word vocabulary and has a significant impact on both training and testing run-times. Several methods have been proposed to cope with this scaling issue by replacing the softmax with a more computationally efficient component at train time. 1 These include importance sampling (Bengio and et al, 2003), hierarchical softmax (Minh and Hinton, 2008), BlackOut (Ji et al., 2016) and Noise Contrastive Estimation (NCE) (Gutmann and Hyvarinen, 2012). NCE has been applied to train neural LMs with large vocabularies (Mnih and Teh, 2012) and more recently was also successfully used to train LSTM-RNN LMs (Vaswani et al., 2013; Chen et al., 2015; Zoph et al., 2016), achieving near state-of- the-art performance on language modeling tasks (Jozefowicz et al., 2016; Chen et al., 2016). All the above works focused on reducing the complexity at train time. However, at test time, the assumption was that one still needs to compute the costly softmax normalization term to obtain a normalized score fit as an estimate for the probability of a word. Self-normalization was recently proposed to address the test time complexity. A self-normalized dis- criminative model is trained to produce near-normalized scores in the sense that the sum over the scores of all words is approximately one. If this approximation is close enough, the assumption is that the costly exact normalization can be waived at test time without significantly sacrificing prediction accuracy …

Content truncated to 500 words