Improving Neural Language Models with Weight Norm Initialization and Regularization

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 93–100 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64010 Improving Neural Language Models with Weight Norm Initialization and Regularization Christian Herold∗ Yingbo Gao∗ Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Hermann Ney Abstract Embedding and projection matrices are com- monly used in neural language models (NLM) as well as in other sequence processing net- works that operate on large vocabularies. We examine such matrices in fine-tuned language models and observe that a NLM learns word vectors whose norms are related to the word frequencies. We show that by initializing the weight norms with scaled log word counts, to- gether with other techniques, lower perplexi- ties can be obtained in early epochs of train- ing. We also introduce a weight norm regular- ization loss term, whose hyperparameters are tuned via a grid search. With this method, we are able to significantly improve perplexities on two word-level language modeling tasks (without dynamic evaluation): from 54.44 to 53.16 on Penn Treebank (PTB) and from 61.45 to 60.13 on WikiText-2 (WT2). 1 Introduction A language model (LM) measures how likely a certain sequence of words is for a given language. It does so by calculating the probability of occur- rence of that sequence, which can be learned from monolingual text data. Many models in machine translation and automatic speech recognition ben- efit from the use of a LM (Corazza et al., 1995; Peter et al., 2017). While count-based LMs (Katz, 1987; Kneser and Ney, 1995) provided the best results in the past, substantial improvements were achieved with the introduction of neural networks in the field of language modeling (Bengio et al., 2003). Different types of architectures such as feedfor- ward neural networks (Schwenk, 2007) and re- current neural networks (Mikolov et al., 2010) have since been used for language modeling. Currently, variants of long short-term memory ∗Equal contribution. Ordering determined by coin flipping. (LSTM) (Hochreiter and Schmidhuber, 1997) net- works give the best results on popular language modeling tasks (Yang et al., 2018). In natural language processing, words are typ- ically represented by high-dimensional one-hot vectors. To reduce dimensionality and to be able to learn relationships between words, they are mapped into a lower-dimensional, continuous em- bedding space. Mathematically, this is done by multiplying the one-hot vector with the embed- ding matrix. Similarly, to receive a probability dis- tribution over the vocabulary, a mapping from an embedding space is performed by a projection ma- trix followed by a softmax operation. These two matrices can be tied together in order to reduce the number of parameters and improve the results of NLMs (Inan et al., 2017; Press and Wolf, 2017). Since the row vectors in the embedding and pro- jection matrices are effectively word vectors in a continuous space, we investigate such weight vec- tors in well-trained and fine-tuned NLMs. We ob- serve that the …

Content truncated to 500 words