MulCode: A Multiplicative Multi-way Model for Compressing Neural Language Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5257–5266, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5257 MulCode: A Multiplicative Multi-way Model for Compressing Neural Language Model Yukun Ma1∗, Pei-Hung(Patrick) Chen2*, Cho-Jui Hsieh2 1AIR Labs, Continental Automotive Group, Singapore 2Department of Computer Science, UCLA Abstract It is challenging to deploy deep neural nets on memory-constrained devices due to the explo- sion of numbers of parameters. Especially, the input embedding layer and Softmax layer usu- ally dominate the memory usage in an RNN- based language model. For example, input embedding and Softmax matrices in IWSLT- 2014 German-to-English data set account for more than 80% of the total model parame- ters. To compress these embedding layers, we propose MulCode, a novel multi-way multi- plicative neural compressor. MulCode learns an adaptively created matrix and its multi- plicative compositions. Together with a prior weighted loss, MulCode is more effective than the state-of-the-art compression methods. On the IWSLT-2014 machine translation data set, MulCode achieved 17 times compression rate for the embedding and Softmax matrices, and when combined with quantization technique, our method can achieve 41.38 times compres- sion rate with very little loss in performance. 1 Introduction Deep neural language models with a large number of parameters have achieved great successes in fa- cilitating a wide range of Natural Language Pro- cessing (NLP) applications. However, the over- whelming size of parameters used by these models stands out as the main obstacle hindering their de- ployment on memory-constrained devices. Given the over-parameterization of deep neural nets, the effective compression of them has been receiving increasing attention from the research community. The neural language models typically consist of three major components: the recurrent layers (e.g., a LSTM cell), an embedding layer for representing input tokens, and a Softmax layer for generating output tokens. The dimension of recurrent layers is typically small and independent of the size of ∗ The two authors contribute equally input/output vocabulary. In contrast, the embed- ding layer and Softmax layer use a dense layer to map the vocabulary to a lower dimensional space, which grows linearly with the vocabulary size and contributes mostly to memory consumption. For example, the One Billion Word language model- ing task (Chelba et al., 2013) has a vocabulary size around 800k, and more than 90% of the memory usage is used to store the input and Softmax matri- ces. Compressing the word embedding, therefore, is the key to reducing the memory usage of deep neural language models. There exist several studies for compressing neu- ral language models. In particular, quantization (Lin et al., 2016; Hubara et al., 2016) and group- wise low rank approximation (Chen et al., 2018a) achieve the state-of-the-art performance on tasks like language modeling and machine translation. However, these methods are rather computation- ally inspired than being motivated by intuitions of the semantic composition and thus suffer from poor interpretability. In contrast, deep composi- tional models …

Content truncated to 500 words