Reusing Weights in Subword-Aware Neural Language Models

Proceedings of NAACL-HLT 2018, pages 1413–1423 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Reusing Weights in Subword-aware Neural Language Models Zhenisbek Assylbekov School of Science and Technology Nazarbayev University zhassylbekov@nu.edu.kz Rustem Takhanov School of Science and Technology Nazarbayev University rustem.takhanov@nu.edu.kz Abstract We propose several ways of reusing subword embeddings and other weights in subword- aware neural language models. The pro- posed techniques do not benefit a competi- tive character-aware model, but some of them improve the performance of syllable- and morpheme-aware models while showing sig- nificant reductions in model sizes. We dis- cover a simple hands-on principle: in a multi- layer input embedding model, layers should be tied consecutively bottom-up if reused at out- put. Our best morpheme-aware model with properly reused weights beats the competitive word-level model by a large margin across multiple languages and has 20%–87% fewer parameters. 1 Introduction A statistical language model (LM) is a model which assigns a probability to a sequence of words. It is used in speech recognition, ma- chine translation, part-of-speech tagging, informa- tion retrieval and other applications. Data sparsity is a major problem in building traditional n-gram language models, which assume that the proba- bility of a word only depends on the previous n words. To deal with potentially severe problems when confronted with any n-grams that have not explicitly been seen before, some form of smooth- ing is necessary. Recent progress in statistical language mod- eling is connected with neural language models (NLM), which tackle the data sparsity problem by representing words as vectors. Typically this is done twice: at input (to embed the current word of a sequence into a vector space) and at output (to embed candidates for the next word of a se- quence). Especially successful are the models in which the architecture of the neural network be- tween input and output is recurrent (Mikolov et al., 2010), which we refer to as recurrent neural net- work language models (RNNLM). Tying input and output word embeddings in word-level RNNLM is a regularization technique, which was introduced earlier (Bengio et al., 2001; Mnih and Hinton, 2007) but has been widely used relatively recently, and there is empirical evidence (Press and Wolf, 2017) as well as theoretical justi- fication (Inan et al., 2017) that such a simple trick improves language modeling quality while de- creasing the total number of trainable parameters almost two-fold, since most of the parameters are due to embedding matrices. Unfortunately, this regularization technique is not directly applicable to subword-aware neural language models as they receive subwords at input and return words at out- put. This raises the following questions: Is it pos- sible to reuse embeddings and other parameters in subword-aware neural language models? Would it benefit language modeling quality? We exper- imented with different subword units, embedding models, and ways of reusing parameters, and our answer to both questions is as follows: There are several ways to reuse weights in subword-aware neural language models, and none of them …

Content truncated to 500 words