Improved Language Modeling by Decoding the Past

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1468–1476 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1468 Improved Language Modeling by Decoding the Past Siddhartha Brahma IBM Research, Almaden, USA brahma@us.ibm.com Abstract Highly regularized LSTMs achieve impressive results on several benchmark datasets in lan- guage modeling. We propose a new regu- larization method based on decoding the last token in the context using the predicted dis- tribution of the next token. This biases the model towards retaining more contextual in- formation, in turn improving its ability to pre- dict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method improves perplexity on the Penn Treebank dataset by up to 1.8 points and by up to 2.3 points on the WikiText-2 dataset, over strong regularized baselines using a single softmax. With a mixture-of-softmax model, we show gains of up to 1.0 perplexity points on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. Each of these results constitute im- provements over models without PDR in their respective settings. 1 Introduction Language modeling is a fundamental task in natu- ral language processing. Given a sequence of to- kens, its joint probability distribution can be mod- eled using the auto-regressive conditional factor- ization. This leads to a convenient formulation where a language model has to predict the next to- ken given a sequence of tokens as context. Recur- rent neural networks are an effective way to com- pute distributed representations of the context by sequentially operating on the embeddings of the tokens. These representations can then be used to predict the next token as a probability distribution over a fixed vocabulary using a linear decoder fol- lowed by Softmax. Starting from the work of (Mikolov et al., 2010), there has been a long list of works that seek to improve language modeling performance using more sophisticated recurrent neural net- works (RNNs) (Zaremba et al., 2014; Zilly et al., 2017; Zoph and Le, 2016; Mujika et al., 2017). However, in more recent work vanilla LSTMs (Hochreiter and Schmidhuber, 1997) with rel- atively large number of parameters have been shown to achieve state-of-the-art performance on several standard benchmark datasets both in word- level and character-level perplexity (Merity et al., 2018b,a; Melis et al., 2018; Yang et al., 2017). A key component in these models is the use of several forms of regularization e.g. varia- tional dropout on the token embeddings (Gal and Ghahramani, 2016), dropout on the hidden-to- hidden weights in the LSTM (Wan et al., 2013), norm regularization on the outputs of the LSTM and classical dropout (Srivastava et al., 2014). By carefully tuning the hyperparameters associated with these regularizers combined with optimiza- tion algorithms like NT-ASGD (a variant of Aver- aged SGD), it is possible to achieve very good per- formance. Each of these regularizations address different parts of the LSTM …

Content truncated to 500 words