Improving Low Compute Language Modeling with In-Domain Embedding Initialisation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8625–8634, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8625 Improving Low Compute Language Modeling with In-Domain Embedding Initialisation Charles Welch, Rada Mihalcea and Jonathan K. Kummerfeld Computer Science & Engineering University of Michigan {cfwelch,mihalcea,jkummerf}@umich.edu Abstract Many NLP applications, such as biomedical data and technical support, have 10-100 mil- lion tokens of in-domain data and limited com- putational resources for learning from it. How should we train a language model in this sce- nario? Most language modeling research con- siders either a small dataset with a closed vo- cabulary (like the standard 1 million token Penn Treebank), or the whole web with byte- pair encoding. We show that for our target set- ting in English, initialising and freezing input embeddings using in-domain data can improve language model performance by providing a useful representation of rare words, and this pattern holds across several different domains. In the process, we show that the standard con- vention of tying input and output embeddings does not improve perplexity when initializing with embeddings trained on in-domain data. 1 Introduction Language modeling is an essential part of many NLP applications, including predictive keyboards, speech recognition, and translation. Recent work has focused on (1) small constrained datasets, such as the Penn Treebank (Marcus et al., 1993) and WikiText-103 (Merity et al., 2017b), and (2) vast resources with billions of words used to train enor- mous models with significant computational re- quirements (Radford et al., 2019). This leaves a gap: when a substantial amount of in-domain data is available, but computational power is limited. We explore how initialising word embeddings using in-domain data can improve language mod- eling in English. Testing all valid configurations of weight tying, embedding freezing, and initialisa- tion, we find that the standard configuration is not optimal when rare words are present. Instead, the best approach is to initialise with in-domain data, untie the input and output, and freeze the input. To understand this difference, we run a series of experiments to measure the impact of changing (a) the threshold for replacing rare words with a spe- cial symbol; (b) the source of data for initialisation; (c) the amount of training data for the language model; and (d) the hyperparameters for both the baseline and our proposed approach. We find that the improvement comes from improved represen- tation of rare words. These findings are confirmed through experiments on four additional domains, with similar trends. We also compare our approach to an n-gram lan- guage model and a large-scale transformer model. We find that if a large-scale transformer is inappro- priate either for computational or modeling reasons, it is best to train an LSTM-based language model with as much data as possible and initialise the embeddings on all available in-domain data. 2 Proposed Approach We propose initialising the language model’s word embeddings with vectors trained on additional in- domain data. To make this most effective, we make two other key changes to …

Content truncated to 500 words