Adaptive Compression of Word Embeddings

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3950–3959 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3950 Adaptive Compression of Word Embeddings Yeachan Kim1 Kang-Min Kim2 SangKeun Lee2,3 1Artificial Intelligence Research Institute, Republic of Korea 2Department of Computer Science and Engineering 3Department of Artificial Intelligence Korea University, Seoul, Republic of Korea yeachan@airi.kr, {kangmin89, yalphy}@korea.ac.kr Abstract Distributed representations of words have been an indispensable component for natural lan- guage processing (NLP) tasks. However, the large memory footprint of word embeddings makes it challenging to deploy NLP mod- els to memory-constrained devices (e.g., self- driving cars, mobile devices). In this pa- per, we propose a novel method to adaptively compress word embeddings. We fundamen- tally follow a code-book approach that repre- sents words as discrete codes such as (8, 5, 2, 4). However, unlike prior works that as- sign the same length of codes to all words, we adaptively assign different lengths of codes to each word by learning downstream tasks. The proposed method works in two steps. First, each word directly learns to select its code length in an end-to-end manner by applying the Gumbel-softmax tricks. After selecting the code length, each word learns discrete codes through a neural network with a bi- nary constraint. To showcase the general ap- plicability of the proposed method, we eval- uate the performance on four different down- stream tasks. Comprehensive evaluation re- sults clearly show that our method is effec- tive and makes the highly compressed word embeddings without hurting the task accuracy. Moreover, we show that our model assigns word to each code-book by considering the sig- nificance of tasks. 1 Introduction Deep neural networks have greatly improved the performance in various tasks, such as image clas- sification (Huang et al., 2017), text classification (Liu and Lapata, 2018), and machine translation (Edunov et al., 2018). This break-through perfor- mance facilitates the demand to deploy such mod- els to embedded systems (e.g., self-driving cars, mobile devices). However, the neural models typi- cally require a large storage or memory footprint, which is a significant concern when deploying neu- ral models to memory-constrained devices (Hinton et al., 2015). To alleviate this limitation, several works have proposed methods that compress the neural models while minimizing loss of accuracy as much as possible (Han et al., 2015, 2016; Liu and Zhu, 2018). However, deploying models for natural language processing (NLP) tasks is challenging. Unlike other domains, NLP models have an embedding layer which maps words and phrases to real-valued vectors. The problem is that these embeddings usu- ally take more parameters than the remaining net- works. In practice, for a neural translation model in OpenNMT (Klein et al., 2017), the word em- bedding parameters accout for 80% of the total parameters. Therefore, it is significant to reduce the parameters of the embedding layer for deploy- ing NLP models to memory-constrained devices. To compress word embeddings, several works proposed code-book based approaches (Shu and Nakayama, 2018; Tissier et al., 2019), which …

Content truncated to 500 words