Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2774–2784 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2774 Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition Vasileios Lioutas1∗, Ahmad Rashid2, Krtin Kumar2, Md. Akmal Haidar2, Mehdi Rezagholizadeh2 1University of British Columbia, 2Huawei Noah’s Ark Lab contact@vlioutas.com, ahmad.rashid@huawei.com, krtin.kumar@huawei.com, md.akmal.haidar@huawei.com, mehdi.rezagholizadeh@huawei.com Abstract Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored. However, they consume a lot of memory which poses a challenge for edge deployment. Embedding matrices, typically, contain most of the param- eters for language models and about a third for machine translation systems. In this pa- per, we propose Distilled Embedding, an (in- put/output) embedding compression method based on low-rank matrix decomposition and knowledge distillation. First, we initialize the weights of our decomposed matrices by learn- ing to reconstruct the full pre-trained word- embedding and then fine-tune end-to-end, em- ploying knowledge distillation on the factor- ized embedding. We conduct extensive ex- periments with various compression rates on machine translation and language modeling, using different data-sets with a shared word- embedding matrix for both embedding and vo- cabulary projection matrices. We show that the proposed technique is simple to replicate, with one fixed parameter controlling compression size, has higher BLEU score on translation and lower perplexity on language modeling com- pared to complex, difficult to tune state-of-the- art methods. 1 Introduction Deep Learning models are the state-of-the-art in NLP, Computer Vision, Speech Recognition and many other fields in Computer Science and Engi- neering. The remarkable deep learning revolution has been built on top of massive amounts of data (both labeled and unlabeled), and faster computa- tion. In NLP, large pre-trained language models like BERT (Devlin et al., 2019) are state-of-the- art on a large number of downstream NLP prob- ∗Work done during an internship at Huawei Noah’s Ark Lab. lems. The largest publicly available language mod- els are trained with hundred of billions of param- eters (Brown et al., 2020). In machine translation the state-of-the-art models have parameters in the order of billions. Data privacy and server cost are some major issues, driving research towards de- ploying these models on edge-devices. However, running these models on edge-devices, faces mem- ory and latency issues due to limitations of the hardware. Thus, there has been considerable in- terest towards research in reducing the memory footprint and faster inference speed for these mod- els (Sainath et al., 2013; Acharya et al., 2019; Shi and Yu, 2018; Jegou et al., 2010; Chen et al., 2018; Winata et al., 2019). The architecture of deep-learning-based lan- guage generation models can be broken down into three components. The first component, repre- sents the embedding, which maps words in the vocabulary to continuous dense vector represen- tations of the words. In language modeling we typically have one dictionary but machine trans- lation has at least two dictionaries correspond- ing to a translation pair. We model these as a single dictionary with a common embedding ma- …

Content truncated to 500 words