Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models

Proceedings of NAACL-HLT 2019, pages 3268–3273 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3268 Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models Yiben Yang, Ji-Ping Wang Department of Statistics Northwestern University Evanston, IL, 60208, USA {yiben.yang,jzwang}@northwestern.edu Doug Downey Department of Electrical Engineering & Computer Science Northwestern University Evanston, IL, 60208, USA {d-downey}@northwestern.edu Abstract Recurrent neural network language models (RNNLM) form a valuable foundation for many NLP systems, but training the models can be computationally expensive, and may take days to train on a large corpus. We ex- plore a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus. In experi- ments with the Billion-Word and Wikitext cor- pora, we show that the technique is effective, and more time-efficient than simply training on a larger sequential corpus. We also intro- duce new strategies for selecting the most in- formative n-grams, and show that these boost efficiency. 1 Introduction Recurrent neural network models of language (RNNLMs) form a foundation for many natural language processing systems. However, the net- works can be expensive to train: training a single model over several million tokens can take hours, and searching through the large hyperparameter space of RNNLMs often entails training and test- ing hundreds of different models. This makes it burdensome to experiment with new RNNLM ar- chitectures on large corpora, or to train RNNLMs for new textual domains. text.RNNLMs are typically trained on sequential In this paper, we investigate how to effi- ciently augment the training of RNNLMs by reg- ularizing the models to match n-gram statistics taken from a much larger corpus. The motivation is that large-corpus n-gram statistics may be in- formative to an RNNLM trained on a smaller se- quential corpus, but unlike RNNLM training, n- gram statistics are inexpensive to compute even over large corpora. Moreover, the statistics only need to be computed once and can be re-used for training many different smaller-corpus RNNLMs. Naively, regularizing an RNNLM to match a given set of n-gram statistics is non-trivial, be- cause the marginal probabilities that n-gram statis- tics represent are not parameters of the RNNLM. In recent work, Noraset et al. (2018) showed that it was possible to regularize an RNNLM to match given n-gram statistics by training the network, when started from a zero state, to match each n- gram probability. However, the regularization ap- proach in that work had tractability limitations— the time cost of the regularization was sufficiently high that using it was inferior to simply training the RNNLM on more sequential text. In this paper, we present an efficient n-gram reg- ularization technique and show that the technique can improve RNNLM training. Our method has three distinctions from previous work that provide efficiency. First, we prioritize regularizing only the n-grams that are most likely to improve the model, by focusing on cases where the RNNLM’s sequential training corpus diverges significantly from the n-gram statistics. Secondly, we regular- …

Content truncated to 500 words