Tensorized Embedding Layers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4847–4860 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4847 Tensorized Embedding Layers Oleksii Hrinchuk1,2,*, Valentin Khrulkov1,4,*, Leyla Mirvakhabova1,*, Elena Orlova1 & Ivan Oseledets1,3 1 Skolkovo Institute of Science and Technology, Moscow, Russia 2 Moscow Institute of Physics and Technology, Moscow, Russia 3 Institute of Numerical Mathematics, RAS, Moscow, Russia 4 Yandex.Research, Moscow, Russia Abstract The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural lan- guage processing. However, when the vocab- ulary is large, the corresponding weight matri- ces can be enormous, which precludes their de- ployment in a limited resource setting. We in- troduce a novel way of parameterizing embed- ding layers based on the Tensor Train decom- position, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evalu- ate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compres- sion ratios for a wide range of architectures, from MLPs to LSTMs and Transformers. 1 Introduction Deep neural networks (DNNs) typically used in natural language processing (NLP) employ large embeddings layers, which map the input words into continuous representations and usually have the form of lookup tables. Despite such simplic- ity, these layers often occupy a large portion of model weights which may cause problems in train- ing and deploying them in a limited resource set- ting. Thus, the compression of large neural net- works and the development of novel lightweight architectures have become essential problems in NLP research. One way to reduce the number of parameters in the trained model is to imply a specific struc- ture on its weight matrices, e.g., assume that they are low-rank or can be well approximated by low- rank tensor networks (Jaderberg et al., 2014). Such approaches are successful at compressing the pre- trained models, but they do not facilitate the train- ing itself. Furthermore, they usually require an *The first three authors contributed equally to this work additional fine-tuning stage to recover the perfor- mance of the original model (Lebedev et al., 2015; Chen et al., 2018a). In this paper, we introduce a new, parameter efficient embedding layer, termed TT–embedding, which can be plugged in into any model and trained end-to-end. The benefits of our compressed TT– layer are twofold. Firstly, instead of storing huge embedding matrix, we store a sequence of much smaller 2-dimensional and 3-dimensional tensors, necessary for reconstructing the required embed- dings, which allows compressing the model signifi- cantly at the cost of a negligible performance drop. Secondly, the overall number of parameters can be relatively small (and constant) during training. The main contributions of our paper are: • We propose to replace a standard dense em- bedding matrix with a novel compactly param- eterized TT–embedding layer. • We provide a theoretical justification of the proposed method from the softmax bottle- neck (Yang …

Content truncated to 500 words