Compressing Pre-trained Language Models by Matrix Decomposition

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884–889 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 884 Compressing Pre-trained Language Models by Matrix Decomposition Matan Ben Noach† and Yoav Goldberg∗‡ ∗Computer Science Department, Bar-Ilan University, Ramat-Gan Israel †Intel AI Lab, Petah-Tikva Israel ‡Allen Institute for Artificial Intelligence matan.ben.noach@intel.com, yoav.goldberg@gmail.com Abstract Large pre-trained language models reach state- of-the-art results on many different NLP tasks when fine-tuned individually; They also come with a significant memory and computational requirements, calling for methods to reduce model sizes (green AI). We propose a two- stage model-compression method to reduce a model’s inference time cost. We first decom- pose the matrices in the model into smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition. This approach has the benefit of reducing the number of parame- ters while preserving much of the information within the model. We experimented on BERT- base model with the GLUE benchmark dataset and show that we can reduce the number of parameters by a factor of 0.4x, and increase in- ference speed by a factor of 1.45x, while main- taining a minimal loss in metric performance. 1 Introduction Deep learning models have been demonstrated to achieve state-of-the-art results, but require large parameter storage and computation. It’s estimated that training a Transformer model with a neural architecture search has a CO2 emissions equiv- alent to nearly five times the lifetime emissions of the average U.S. car, including its manufactur- ing (Strubell et al., 2019). Alongside the increase in deep learning models complexity, in the NLP domain, there has been a shift in the NLP mod- eling paradigm from training a randomly initial- ized model to fine-tuning a large and computa- tional heavy pre-trained language model (Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2018; Radford, 2018; Radford et al., 2019; Dai et al., 2019; Yang et al., 2019; Lample and Con- neau, 2019; Liu et al., 2019b; Raffel et al., 2019; Lan et al., 2019; Lewis et al., 2019). While re-using pre-trained models offsets the training costs, inference time costs of the fine- tuned models remain significant, and are show- stoppers in many applications. The main chal- lenge with pre-trained models is how can we re- duce their size while saving the information con- tained within them. Recent work, approached this by keeping some of the layers while removing oth- ers (Sanh et al., 2019; Sun et al., 2019; Xu et al., 2020). A main drawback of such approach is in its coarse-grained nature: removing entire lay- ers might discard important information contained within the model, and working at the granularity of layers makes the trade-off between compression and accuracy of a model hard to control. Moti- vated by this, in this work we suggest a more fine- grained approach which decomposes each matrix to two smaller matrices and then perform feature …

Content truncated to 500 words